CUDA driver version is insufficient for CUDA runtime version with nvidia driver 390

Hi there,

I’m running Ubuntu 18.04.
I installed CUDA 9.0 (with the runfile) and CudNN 7.0.5.
My nvidia GPU driver is 390. The minimum driver version should be 384 with CUDA 9.0.

I installed tensorflow-gpu 1.10 but when I try to use the GPU, I get this strange error:

>>> from tensorflow.python.client import device_lib
>>> 
>>> device_lib.list_local_devices()
2018-10-29 18:21:43.976046: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2018-10-29 18:21:44.115646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:65:00.0
totalMemory: 10.91GiB freeMemory: 10.48GiB
2018-10-29 18:21:44.115671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/edge34/miniconda3/envs/inference/lib/python3.6/site-packages/tensorflow/python/client/device_lib.py", line 41, in list_local_devices
    for s in pywrap_tensorflow.list_devices(session_config=session_config)
  File "/home/edge34/miniconda3/envs/inference/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 1711, in list_devices
    return ListDevices(status)
  File "/home/edge34/miniconda3/envs/inference/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 526, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

I tried several times to reinstall cuda and the nvidia drivers. I also reinstalled Ubuntu 18.04 to start from a clean OS. But that brought no improvement.

Some more details about my config:

$ lspci -v | grep -i nvidia
65:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
65:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
$ nvidia-settings -q NvidiaDriverVersion
Attribute 'NvidiaDriverVersion' (edg34:1.0): 390.87
Attribute 'NvidiaDriverVersion' (edg34:1[gpu:0]): 390.87
$ uname -r
4.15.0-29-generic
$ cat /proc/driver/nvidia/version 
NVRM version: NVIDIA UNIX x86_64 Kernel Module  390.87  Tue Aug 21 12:33:05 PDT 2018
GCC version:  gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
$ lsmod | grep -i nvidia
nvidia_uvm            757760  2
nvidia_drm             40960  2
nvidia_modeset       1114112  19 nvidia_drm
nvidia              14364672  1064 nvidia_modeset,nvidia_uvm
drm_kms_helper        172032  1 nvidia_drm
drm                   401408  5 nvidia_drm,drm_kms_helper
ipmi_msghandler        53248  2 nvidia,ipmi_devintf
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
$ nvidia-smi
Mon Oct 29 18:45:40 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.87                 Driver Version: 390.87                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:65:00.0  On |                  N/A |
| 20%   46C    P2    57W / 250W |    476MiB / 11175MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1018      G   /usr/lib/xorg/Xorg                            18MiB |
|    0      1138      G   /usr/bin/gnome-shell                          70MiB |
|    0      1462      G   /usr/lib/xorg/Xorg                           114MiB |
|    0      1608      G   /usr/bin/gnome-shell                         106MiB |
|    0      2060      G   /usr/lib/firefox/firefox                       2MiB |
|    0      2225      G   /usr/lib/firefox/firefox                       2MiB |
|    0      2283      G   /usr/lib/firefox/firefox                       2MiB |
|    0      2356      G   /usr/lib/firefox/firefox                       2MiB |
|    0      2771      C   python                                       151MiB |
+-----------------------------------------------------------------------------+
$ cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 5
--
#define CUDNN_VERSION    (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"

How did you install the 390.87 GPU driver? That’s not part of any CUDA 9.0 runfile installer that I am aware of.

Also note that Ubuntu 18.04 is not an officially supported platform for CUDA 9.0

Did you validate your CUDA install (before trying to use it with TF) using the method contained in the linux install guide for CUDA?

Why don’t you suggest some good combination?
Why ask them like that? because we don’t know, many things need to search without suggest link at download NVIDIA driver or NVIDIA Toolkit.

Nvidia instructions is always late than version and hard to search. - My opinion.

If you work as NVIDIA MODERATOR, please give out your suggestion, we can reinstall system if need. Thankyou

my suggestion is to do a fresh load of whatever operating system you are using, get your installers from http://www.nvidia.com/getcuda, and follow the instructions in the install guide carefully:

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
(that one is for CUDA 10. If you want to install CUDA 9, follow the install guide for CUDA 9)

For example, if you install CUDA 9 on Ubuntu 18.04, you are not following the instructions in the install guide carefully.

I’m not going to provide any suggestion for how to install CUDA 9 on Ubuntu 18.04. It’s not recommended by NVIDIA. If you want to do that, you will have to figure it out.