Could not load dynamic library 'libcudart.so.10.1';

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I have a docker which was built from this dockerfile:

FROM nvidia/cuda

RUN apt-get update \
  && apt-get install -y python3-pip python3-dev \
  && cd /usr/local/bin \
  && ln -s /usr/bin/python3 python \
  && pip3 install --upgrade pip

COPY requirement.txt /tmp/requirement.txt
WORKDIR /tmp
RUN pip install -r requirement.txt

requirement.txt contains (the important parts):

Keras==2.4.3
Keras-Preprocessing==1.1.2
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow-estimator==2.3.0
tensorflow-gpu==2.3.1

When I run nvidia-smi on the running machine I’m getting:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080    Off  | 00000000:01:00.0  On |                  N/A |
| 32%   28C    P8    16W / 215W |    740MiB /  7973MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       473      G   /usr/bin/gnome-shell                         281MiB |
|    0      1726      G   /usr/lib/xorg/Xorg                            18MiB |
|    0      2094      G   /usr/bin/gnome-shell                          77MiB |
|    0     30481      G   ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files   146MiB |
|    0     32763      G   /usr/lib/xorg/Xorg                           212MiB |

I run my docker and I’m getting the following errors:

Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64

2020-11-30 09:10:10.385750: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.ing errors:
3

Hi @amitliron,
It looks like your libraries are not found. You have problems with CUDA libraries.
I guess you must enable the correct CUDA version
Make LD_LIBRARY_PATH point to the correct libraries.
You may check the below doc for reference.

Thanks!

1 Like