Hi there,
I’m running Ubuntu 18.04.
I installed CUDA 9.0 (with the runfile) and CudNN 7.0.5.
My nvidia GPU driver is 390. The minimum driver version should be 384 with CUDA 9.0.
I installed tensorflow-gpu 1.10 but when I try to use the GPU, I get this strange error:
>>> from tensorflow.python.client import device_lib
>>>
>>> device_lib.list_local_devices()
2018-10-29 18:21:43.976046: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2018-10-29 18:21:44.115646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:65:00.0
totalMemory: 10.91GiB freeMemory: 10.48GiB
2018-10-29 18:21:44.115671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/edge34/miniconda3/envs/inference/lib/python3.6/site-packages/tensorflow/python/client/device_lib.py", line 41, in list_local_devices
for s in pywrap_tensorflow.list_devices(session_config=session_config)
File "/home/edge34/miniconda3/envs/inference/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 1711, in list_devices
return ListDevices(status)
File "/home/edge34/miniconda3/envs/inference/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 526, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
I tried several times to reinstall cuda and the nvidia drivers. I also reinstalled Ubuntu 18.04 to start from a clean OS. But that brought no improvement.
Some more details about my config:
$ lspci -v | grep -i nvidia
65:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
65:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
$ nvidia-settings -q NvidiaDriverVersion
Attribute 'NvidiaDriverVersion' (edg34:1.0): 390.87
Attribute 'NvidiaDriverVersion' (edg34:1[gpu:0]): 390.87
$ uname -r
4.15.0-29-generic
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 390.87 Tue Aug 21 12:33:05 PDT 2018
GCC version: gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
$ lsmod | grep -i nvidia
nvidia_uvm 757760 2
nvidia_drm 40960 2
nvidia_modeset 1114112 19 nvidia_drm
nvidia 14364672 1064 nvidia_modeset,nvidia_uvm
drm_kms_helper 172032 1 nvidia_drm
drm 401408 5 nvidia_drm,drm_kms_helper
ipmi_msghandler 53248 2 nvidia,ipmi_devintf
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
$ nvidia-smi
Mon Oct 29 18:45:40 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.87 Driver Version: 390.87 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:65:00.0 On | N/A |
| 20% 46C P2 57W / 250W | 476MiB / 11175MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1018 G /usr/lib/xorg/Xorg 18MiB |
| 0 1138 G /usr/bin/gnome-shell 70MiB |
| 0 1462 G /usr/lib/xorg/Xorg 114MiB |
| 0 1608 G /usr/bin/gnome-shell 106MiB |
| 0 2060 G /usr/lib/firefox/firefox 2MiB |
| 0 2225 G /usr/lib/firefox/firefox 2MiB |
| 0 2283 G /usr/lib/firefox/firefox 2MiB |
| 0 2356 G /usr/lib/firefox/firefox 2MiB |
| 0 2771 C python 151MiB |
+-----------------------------------------------------------------------------+
$ cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 5
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
#include "driver_types.h"