Hi, I am using a standardi dockerfile from nvidia, so my dockerfile starts like this:
FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu16.04
which should install cuda 10.0. I also have a couple flags in the dockerfile to install the specific keras and tensorflow-gpu versions I want:
RUN pip install keras==2.2.4 tensorflow-gpu==1.11.0
however, after the container is built and I try to run it, I get this error:
Traceback (most recent call last):
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py”, line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py”, line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py”, line 24, in swig_import_helper
_mod = imp.load_module(‘_pywrap_tensorflow_internal’, fp, pathname, description)
File “/root/miniconda3/lib/python3.6/imp.py”, line 243, in load_module
return load_dynamic(name, filename, file)
File “/root/miniconda3/lib/python3.6/imp.py”, line 343, in load_dynamic
return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/arbiter.py”, line 583, in spawn_worker
worker.init_process()
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/workers/base.py”, line 129, in init_process
self.load_wsgi()
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/workers/base.py”, line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/app/base.py”, line 67, in wsgi
self.callable = self.load()
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py”, line 52, in load
return self.load_wsgiapp()
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py”, line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File “/root/miniconda3/lib/python3.6/site-packages/gunicorn/util.py”, line 350, in import_app
import(module)
File “/app/inference/common_run.py”, line 1, in
from app import app
File “/app/inference/app/init.py”, line 7, in
from app.inference_classification import inference_classification
File “/app/inference/app/inference_classification.py”, line 6, in
import keras
File “/root/miniconda3/lib/python3.6/site-packages/keras/init.py”, line 3, in
from . import utils
File “/root/miniconda3/lib/python3.6/site-packages/keras/utils/init.py”, line 6, in
from . import conv_utils
File “/root/miniconda3/lib/python3.6/site-packages/keras/utils/conv_utils.py”, line 9, in
from … import backend as K
File “/root/miniconda3/lib/python3.6/site-packages/keras/backend/init.py”, line 89, in
from .tensorflow_backend import *
File “/root/miniconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py”, line 5, in
import tensorflow as tf
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/init.py”, line 22, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/init.py”, line 49, in
from tensorflow.python import pywrap_tensorflow
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py”, line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py”, line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py”, line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File “/root/miniconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py”, line 24, in swig_import_helper
_mod = imp.load_module(‘_pywrap_tensorflow_internal’, fp, pathname, description)
File “/root/miniconda3/lib/python3.6/imp.py”, line 243, in load_module
return load_dynamic(name, filename, file)
File “/root/miniconda3/lib/python3.6/imp.py”, line 343, in load_dynamic
return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory
with the main part of the error, I assume, being the “ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory” part. I’m not sure why it’s looking for cuda 9.0 when the original container is clearly pulled from the cuda:10.0 default? If I run nvidia-smi on the machine that the container is supposed to run on, it looks fine:
±----------------------------------------------------------------------------+
| NVIDIA-SMI 410.78 Driver Version: 410.78 CUDA Version: 10.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000E7D7:00:00.0 Off | Off |
| N/A 46C P0 52W / 149W | 0MiB / 12206MiB | 1% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+
so I’m not sure what’s going wrong here. Any help would be much appreciated!
Note - on a seperate non-docker GPU box that the GPU is working for, I have keras=2.2.4, tensorflow-gpu=1.11.0, and cuda=10.0 which is why I think that combination should work here as well.