CUDA 10.2 & Tensorflow 2.0. Getting an error when testing Tensorflow

Accelerated Computing CUDA CUDA Setup and Installation
1

I installed Cuda 10.2 with the associated cudnn with no issues. Installed Tensorflow 2.0. It doesn’t look its recognizing the GPU.

I know with Cuda 10, I won’t have an issue with Tensorflow 2.0

The big question, I am trying to answer. Does Tensorflow 2.0 work with Cuda 10.2.

>>> import tensorflow as tf
>>>  tf.test.is_gpu_available()
  File "<stdin>", line 1
    tf.test.is_gpu_available()
    ^
IndentationError: unexpected indent
>>>

tarik-deeplearning@tarikdeeplearning-All-Series:~$ nvidia-smi
Sat Dec 28 21:21:26 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 00000000:01:00.0  On |                  N/A |
| 23%   39C    P8    15W / 250W |    275MiB / 12194MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:02:00.0 Off |                  N/A |
| 23%   30C    P8     9W / 250W |      2MiB / 12196MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1338      G   /usr/lib/xorg/Xorg                            18MiB |
|    0      1414      G   /usr/bin/gnome-shell                          49MiB |
|    0      1622      G   /usr/lib/xorg/Xorg                           108MiB |
|    0      1766      G   /usr/bin/gnome-shell                          95MiB |
+-----------------------------------------------------------------------------+

kdeeplearning-All-Series:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
2

Just adding this. & kind of progressing
I am able to see the GPU devices here

>>> import tensorflow as tf
>>> from tensorflow.python.client import device_lib
>>> print(device_lib.list_local_devices())
2019-12-28 22:04:51.491137: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-28 22:04:51.521296: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3298250000 Hz
2019-12-28 22:04:51.522539: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x51538a0 executing computations on platform Host. Devices:
2019-12-28 22:04:51.522584: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-12-28 22:04:51.526595: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-28 22:04:51.841513: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x51e9110 executing computations on platform CUDA. Devices:
2019-12-28 22:04:51.841562: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): TITAN Xp, Compute Capability 6.1
2019-12-28 22:04:51.841586: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): TITAN X (Pascal), Compute Capability 6.1
2019-12-28 22:04:51.846348: I te

However when I test again Tensorflow with following command…Looks like issues with loading specific libraries within lib64. Those libraries are related to Cuda 10.0. Example: ‘libcudart.so.10.0’ Doesn’t exist should be 10.2…Please let me know what you think. Do you have a solution? Build Tf from source may be a solution. For now I am going to have to install cuda 10.0

tf.test.is_gpu_available()
2019-12-28 22:08:23.329723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:02:00.0
2019-12-28 22:08:23.331713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties: 
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
2019-12-28 22:08:23.331944: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2019-12-28 22:08:23.332091: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2019-12-28 22:08:23.332226: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2019-12-28 22:08:23.332355: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2019-12-28 22:08:23.332484: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2019-12-28 22:08:23.332614: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2019-12-28 22:08:23.332653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-28 22:08:23.332672: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-12-28 22:08:23.332773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-28 22:08:23.332796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 1 
2019-12-28 22:08:23.332811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N Y 
2019-12-28 22:08:23.332825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1:   Y N 
False
3

Tensorflow 2.0 is build with CUDA10.1 not CUDA 10.2 see Kaynaktan oluştur  |  TensorFlow
You either have to downgrade your CUDA version or to compile tensorflow without any guaranty it will work with CUDA2.2

Welcome to the world of driver mess !

Have a nice day

JPM

4

I’m not sure what you mean by CUDA 2.1 and CUDA 2.2

Here is what I see in your link:

GPU
Version	                Python version	Compiler	Build tools	cuDNN	CUDA
tensorflow-2.1.0	2.7, 3.3-3.7	GCC 7.3.1	Bazel 0.26.1	7.6	10.1
tensorflow-2.0.0	2.7, 3.3-3.7	GCC 7.3.1	Bazel 0.26.1	7.4	10.0
5

Sorry, I made a mess myself I was talking about CUDA 10.1 ans CUDA 10.2

6

I also have CUDA 10.2. Does exist a TensorFlow version, compatible with CUDA 10.2 ?

7

I’ve switch to archlinux instead of ubuntu and all is fine and uptodate

Le lun. 16 mars 2020 à 14:31, py.ohayo via NVIDIA Developer Forums nvidia@discoursemail.com a écrit :

8

I have cuda 10.2 and tensorflow 2.1 working. The only thing is that all your calls to ‘keras…’ now have to be ‘tensorflow.keras…’