Failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error : Ubuntu 20.04.2, RTX 2070 SUPER GPU

Graphics / Linux Linux
1

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.2 LTS
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version: 2.5.0
  • Python version: 3.7.10
  • Installed using virtualenv? pip? conda?: pip
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source): 9.3.0
  • CUDA/cuDNN version: 11.3
  • GPU model and memory: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] / 8GB

I have been receiving the E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error for running the simplest tensorflow command of creating a constant. The command executes successfully, but I could see tensorflow using only CPU for this.

I have also checked this by a running a simple script that compares the time of execution of a compute problem between GPU and CPU in which case same error message and also the time of computation for both CPU and GPU came out to be same. I feel tensorflow is not able to detect the GPU in this system.

The tensorflow and CUDA installation were based on the instructions provided in this link which are follows,

 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin

sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600

 wget https://developer.download.nvidia.com/compute/cuda/11.3.1/local_installers/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.debsudo dpkg -i cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb

 sudo apt-key add /var/cuda-repo-ubuntu2004-11-3-local/7fa2af80.pub

 sudo apt-get update

 sudo apt-get -y install cuda

I have used the following instructions to install cudnn:

sudo dpkg -i libcudnn8_8.2.0.53-1+cuda11.3_amd64.deb
sudo dpkg -i libcudnn8-dev_8.2.0.53-1+cuda11.3_amd64.deb

Following is the snapshot of output I get from creating the tensorflow constant variable,

(tkeras) smart@smart-B460MDS3H:/usr/local$ python
Python 3.7.10 (default, Feb 26 2021, 18:47:35) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-05-29 10:47:51.427226: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
>>> a = tf.constant(20)
2021-05-29 10:47:55.936327: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-05-29 10:47:55.959954: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2021-05-29 10:47:55.959977: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: smart-B460MDS3H
2021-05-29 10:47:55.959982: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: smart-B460MDS3H
2021-05-29 10:47:55.960023: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.73.1
2021-05-29 10:47:55.960041: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.73.1
2021-05-29 10:47:55.960046: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.73.1
2021-05-29 10:47:55.960242: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> tf.__version__
'2.5.0'

I am also attaching the gpu versus cpu time comparison script with this issue tempScript.txt

1 Like