Trouble with tensorflow_20.01-tf2-py3 image and singularity


I’m using tensorflow_20.01-tf2-py3 to build an image for development of Tensorflow-based classification and regression algorithms. I’m having an interesting issue. When I start the baseline image (not even extending it yet) with docker, everything works fine and I can run Numpy and Tensorflow as expected.

But when I use singularity:

$ singularity exec tensorflow_20.01-tf2-py3.sif  /bin/bash
$ python3
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/numpy/", line 142, in <module>
    from . import core
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/", line 67, in <module>
    raise ImportError(msg.format(path))
ImportError: Something is wrong with the numpy installation. While importing we detected an older version of numpy in ['/usr/local/lib/python3.6/dist-packages/numpy']. One method of fixing this is to repeatedly uninstall numpy until none is found, then reinstall this version.

Somehow the image, when run in singularity, has issues with loading numpy (and therefore TF as well). I’m at a bit of a loss on what to do: from all my exploring of the filesystem, I can only see a single numpy installation at /usr/local/lib/python3.6/dist-packages/numpy.

I’ve tried running this on multiple different machines, ranging from an Ubuntu 18.04 machine, to a CentOS7 machine, to my local OSX 10.15 laptop, and I get the same error regardless of the host.

This also seems to be broken in 19.12, but was fine in 19.11.

Can anyone provide any tips on how to get around this? Thanks for your help!

Ok, I found a workaround… in my Dockerfile I added:

RUN python3 -m pip uninstall -y numpy && python3 -m pip uninstall -y numpy && python3 -m pip uninstall -y numpy

RUN python3 -m pip install numpy

In the past, I thought that one uninstall was all that was needed, since the docker build gives a warning that the second and third don’t do anything. But with only one uninstall, the ghost numpy was still around.

Anyway this seems to work for me now, but I hope the image can be centrally fixed so others don’t run into this when using singularity.