Unable to use GPU with Tensorflow 2.1 + CUDA 10.1 on Ubuntu 18.04

hue · February 8, 2020, 4:54pm

I’ve tried reading other solutions but my problem seems different.
I’ve followed the guide at GPU 支持 | TensorFlow (for the most part).

Running the command (in venv)

python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

outputs

$ python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2020-02-08 11:43:21.944657: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-02-08 11:43:21.946221: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvrtc.so.10.1: cannot open shared object file: No such file or directory
2020-02-08 11:43:21.946258: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-02-08 11:43:22.977924: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-02-08 11:43:23.008777: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-08 11:43:23.009295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 298.32GiB/s
2020-02-08 11:43:23.009334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-08 11:43:23.009360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-08 11:43:23.009446: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2020-02-08 11:43:23.009488: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2020-02-08 11:43:23.009525: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-02-08 11:43:23.009570: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2020-02-08 11:43:23.009592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-08 11:43:23.009597: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-02-08 11:43:23.009968: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-08 11:43:23.039141: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3593135000 Hz
2020-02-08 11:43:23.040033: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5b3fad0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-08 11:43:23.040055: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-02-08 11:43:23.123931: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-08 11:43:23.124538: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5bc5a20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-02-08 11:43:23.124570: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
2020-02-08 11:43:23.124683: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-08 11:43:23.124700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      
tf.Tensor(317.43338, shape=(), dtype=float32)

Clearly I’m missing

load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such
load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No
load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No
load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No

nvidia-smi is

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
| 28%   27C    P8     6W / 180W |    256MiB /  8118MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

l /usr/local/cuda-10.1/lib64/

outputs

libcudadevrt.a  libcudart.so.10.1@     libcudart_static.a  libOpenCL.so@    libOpenCL.so.1.1
libcudart.so@   libcudart.so.10.1.243  libculibos.a        libOpenCL.so.1@  stubs/

Additional ask: why is

Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvrtc.so.10.1: cannot open shared object file

not found for TensorRT?

hue · February 11, 2020, 2:09am

I’m looking to use GPU by the way. The reason I purchased it was to be able to train on it.

hue · February 19, 2020, 1:12am

Got 'r!

I haven’t repeated these steps because it’s working and there’s no need for fixin.

Install the latest nvidia driver (was nvidia-driver-440 for me) according to https://www.linuxbabe.com/ubuntu/install-nvidia-driver-ubuntu-18-04 "...From the Command Line" and reboot
Run ``` nvidia-smi ``` and note the version of CUDA (was 10.2 for me).
Install CUDA after selecting the matching environment from https://developer.nvidia.com/cuda-downloads?target_os=Linux
Install tensorflow 2 according to https://www.tensorflow.org/install/pip. Remember virtualenv is installed with pip3.
In virtualenv run the python code for the number of gpus according to https://www.tensorflow.org/guide/gpu. Had two errors 1. Related to libnvinfer 2. Related to libcudart.so.10.1
For the libnvinfer had to take actions to install TensorRT. I didn't take notes on how to resolve this but I had to run a few commands to fix and install. Also I remember using the TensorRT install from https://www.tensorflow.org/install/gpu
For libcudart.so.10.1 I created a soft link at /usr/local/cuda/lib64 similar to what was created for libcudart.so.10.2. and included /usr/local/cuda/lib64/ in LD_LIBRARY_PATH

Running the number of gpus python script gives me nothing but success

2020-02-18 20:10:45.642047: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-02-18 20:10:45.643266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-02-18 20:10:46.116436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-02-18 20:10:46.143017: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-18 20:10:46.143624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 298.32GiB/s
2020-02-18 20:10:46.143702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-18 20:10:46.143729: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-18 20:10:46.144988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-18 20:10:46.145255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-18 20:10:46.146517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-18 20:10:46.147202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-18 20:10:46.147231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-18 20:10:46.147339: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-18 20:10:46.148088: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-18 20:10:46.148512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
Num GPUs Available:  1

If at first you don’t succeed. Take a break and like time through the hourglass…

Topic		Replies	Views
Tensorflow 2.1 with CUDA10.2 warnings .. Frameworks (archived) tensorflow	15	17793	July 3, 2020
Error "Could not load dynamic library" for Tensorflow 2.1.0 with Cuda 10.1 Frameworks (archived) tensorflow	1	3680	March 31, 2020
Cuda and tensorflow CUDA Developer Tools	0	1137	September 18, 2020
TensorFlow - GPU installation issue Computer Vision & Image Processing tensorflow	6	16519	March 3, 2023
Tensorflow fails to find libcudart CUDA on Windows Subsystem for Linux	7	18783	September 23, 2020
all CUDA-capable devices are busy or unavailable. What is wrong? cuDNN	10	9799	October 12, 2021
Nvidia process not running Linux	25	2898	December 31, 2021
Could not load dynamic library 'libcudart.so.10.1'; TensorRT	1	8737	November 30, 2020
CUDA 10.2 & Tensorflow 2.0. Getting an error when testing Tensorflow CUDA Setup and Installation	7	20978	March 20, 2020
Nano with jetpack 4.3 can't find gpu with tensorflow 2.1 Jetson Nano tensorflow	9	1472	October 18, 2021

Unable to use GPU with Tensorflow 2.1 + CUDA 10.1 on Ubuntu 18.04

Related topics