- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.2 LTS
- TensorFlow installed from (source or binary): binary
- TensorFlow version: 2.5.0
- Python version: 3.7.10
- Installed using virtualenv? pip? conda?: pip
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source): 9.3.0
- CUDA/cuDNN version: 11.3
- GPU model and memory: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] / 8GB
I have been receiving the E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error for running the simplest tensorflow command of creating a constant. The command executes successfully, but I could see tensorflow using only CPU for this.
I have also checked this by a running a simple script that compares the time of execution of a compute problem between GPU and CPU in which case same error message and also the time of computation for both CPU and GPU came out to be same. I feel tensorflow is not able to detect the GPU in this system.
The tensorflow and CUDA installation were based on the instructions provided in this link which are follows,
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/11.3.1/local_installers/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.debsudo dpkg -i cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb sudo apt-key add /var/cuda-repo-ubuntu2004-11-3-local/7fa2af80.pub sudo apt-get update sudo apt-get -y install cuda
I have used the following instructions to install cudnn:
sudo dpkg -i libcudnn8_18.104.22.168-1+cuda11.3_amd64.deb sudo dpkg -i libcudnn8-dev_22.214.171.124-1+cuda11.3_amd64.deb
Following is the snapshot of output I get from creating the tensorflow constant variable,
(tkeras) smart@smart-B460MDS3H:/usr/local$ python Python 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf 2021-05-29 10:47:51.427226: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 >>> a = tf.constant(20) 2021-05-29 10:47:55.936327: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2021-05-29 10:47:55.959954: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error 2021-05-29 10:47:55.959977: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: smart-B460MDS3H 2021-05-29 10:47:55.959982: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: smart-B460MDS3H 2021-05-29 10:47:55.960023: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.73.1 2021-05-29 10:47:55.960041: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.73.1 2021-05-29 10:47:55.960046: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.73.1 2021-05-29 10:47:55.960242: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. >>> tf.__version__ '2.5.0'
I am also attaching the gpu versus cpu time comparison script with this issue tempScript.txt