Error: CUDA driver version is insufficient for CUDA runtime version

I’m trying to verify that cuDNN is installed and is running properly following instructions on http://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html. I get the following error:

kimmo@kimmo-System-Product-Name:~/cudnn_samples_v7/mnistCUDNN$ ./mnistCUDNN
cudnnGetVersion() : 7005 , CUDNN_VERSION from cudnn.h : 7005 (7.0.5)
Cuda failurer version : GCC 5.4.0
Error: CUDA driver version is insufficient for CUDA runtime version
error_util.h:93
Aborting…

I tried uninstalling and reinstalling the driver as follows:
sudo apt-get purge nvidia-*
sudo apt-get install nvidia-384

I still get the same error.

nvidia-smi shows the following:

kimmo@kimmo-System-Product-Name:~/cudnn_samples_v7/mnistCUDNN$ nvidia-smi
Wed Dec 27 10:50:27 2017
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90 Driver Version: 384.90 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106… Off | 00000000:01:00.0 On | N/A |
| 0% 46C P8 10W / 200W | 241MiB / 6069MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 963 G /usr/lib/xorg/Xorg 198MiB |
| 0 1327 G compiz 40MiB |
±----------------------------------------------------------------------------+

Any help would be greatly appreciated.

BR, Kimmo

  1. get your cuda installers here: http://www.nvidia.com/getcuda
  2. follow the instructions in the linux install guide, carefully, including verification: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
  3. Then install CUDNN

You don’t indicate which CUDA version you have attempted to install, but if you are attempting to install CUDA 9.1 (latest, currently) then it will not work with r384 driver (384.xx). It requires a r387 driver (387.xx) and if you install CUDA properly, you will get a r387 driver.

I uninstalled CUDA and followed instructions to install CUDA9.1 (this time hopefully more carefully). Following the post-installation actions I’m supposed to create a script in /usr/lib/systemd/system/. However, no such folder exists in my Ubuntu 16.04 environment. Can I use another folder, perhaps /etc/systemd/system/?

Are you working on a Power9 system?

It seems unlikely to me since your GPU is a GeForce GTX 1060. The folder/script you are referring to is only listed in the section “Power9 setup”

Why are you doing that?

Sorry, I missed the header. No, I’m not working on a Power9 system.

I ran into other problems. When I was trying to verify the installation, I got:

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL

I checked the driver and it was still 384.90:

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 384.90 Tue Sep 19 19:17:35 PDT 2017
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)

I thought that I might have to update the driver manually and ran:

$ sudo apt-get purge nvidia-*

That was probably not a good idea, since I still get:

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 384.90 Tue Sep 19 19:17:35 PDT 2017
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)

And now I also get:

deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

What should I do?

I guess you’re not following the instructions I indicated.

I noticed that the checksum of the installer I downloaded is different than in https://developer.download.nvidia.com/compute/cuda/9.1/Prod/docs/sidebar/md5sum.txt

d41d8cd98f00b204e9800998ecf8427e cuda-repo-ubuntu1604-9-1-local_9.1.85-1_amd64.deb (this the one I downloded)
e6bd313b318660dc3b4aa30481c2d406 cuda-repo-ubuntu1604-9-1-local_9.1.85-1_amd64.deb (this is the one on the site)

I tried downloading the installer multiple times but I always get the same checksum. What does this mean?

I finally manged to make the cuda and cudnn installations work. I found no other way to resolve the issue with the nvida driver but to reinstall Ubuntu. After reinstallation I followed the cuda installation instructions carefully and passed.

Installing cudnn I encountered the below issue:

$ make clean && make
rm -rf *o
rm -rf mnistCUDNN
/usr/local/cuda/bin/nvcc -ccbin g++ -I/usr/local/cuda/include -IFreeImage/include -m64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_53,code=compute_53 -o fp16_dev.o -c fp16_dev.cu
g++ -I/usr/local/cuda/include -IFreeImage/include -o fp16_emu.o -c fp16_emu.cpp
g++ -I/usr/local/cuda/include -IFreeImage/include -o mnistCUDNN.o -c mnistCUDNN.cpp
In file included from /usr/local/cuda/include/channel_descriptor.h:62:0,
from /usr/local/cuda/include/cuda_runtime.h:90,
from /usr/include/cudnn.h:64,
from mnistCUDNN.cpp:30:
/usr/local/cuda/include/cuda_runtime_api.h:1683:101: error: use of enum ‘cudaDeviceP2PAttr’ without previous declaration
__ cudaError_t CUDARTAPI cudaDeviceGetP2PAttribute(int *value, enum cudaDeviceP
^
/usr/local/cuda/include/cuda_runtime_api.h:2920:102: error: use of enum ‘cudaFuncAttribute’ without previous declaration
_ cudaError_t CUDARTAPI cudaFuncSetAttribute(const void *func, enum cudaFuncAtt
^
In file included from /usr/local/cuda/include/channel_descriptor.h:62:0,
from /usr/local/cuda/include/cuda_runtime.h:90,
from /usr/include/cudnn.h:64,
from mnistCUDNN.cpp:30:
/usr/local/cuda/include/cuda_runtime_api.h:5763:92: error: use of enum ‘cudaMemoryAdvise’ without previous declaration
or_t CUDARTAPI cudaMemAdvise(const void *devPtr, size_t count, enum cudaMemoryA
^
/usr/local/cuda/include/cuda_runtime_api.h:5820:98: error: use of enum ‘cudaMemRangeAttribute’ without previous declaration
UDARTAPI cudaMemRangeGetAttribute(void *data, size_t dataSize, enum cudaMemRang
^
/usr/local/cuda/include/cuda_runtime_api.h:5857:102: error: use of enum ‘cudaMemRangeAttribute’ without previous declaration
TAPI cudaMemRangeGetAttributes(void **data, size_t *dataSizes, enum cudaMemRang
^
Makefile:200: recipe for target ‘mnistCUDNN.o’ failed

I managed to resolve it opening the file:
/usr/include/cudnn.h

…And changing the line:
#include “driver_types.h”

…to:
#include <driver_types.h>

This resolution was suggested in https://devtalk.nvidia.com/default/topic/1006726/cudnnv6-mnist-example-compile-errors/

I finally manged to make the cuda and cudnn installations work. I found no other way to resolve the issue with the nvida driver but to reinstall Ubuntu. After reinstallation I followed the cuda installation instructions carefully and passed.

Installing cudnn I encountered the below issue:

$ make clean && make
rm -rf *o
rm -rf mnistCUDNN
/usr/local/cuda/bin/nvcc -ccbin g++ -I/usr/local/cuda/include -IFreeImage/include -m64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_53,code=compute_53 -o fp16_dev.o -c fp16_dev.cu
g++ -I/usr/local/cuda/include -IFreeImage/include -o fp16_emu.o -c fp16_emu.cpp
g++ -I/usr/local/cuda/include -IFreeImage/include -o mnistCUDNN.o -c mnistCUDNN.cpp
In file included from /usr/local/cuda/include/channel_descriptor.h:62:0,
from /usr/local/cuda/include/cuda_runtime.h:90,
from /usr/include/cudnn.h:64,
from mnistCUDNN.cpp:30:
/usr/local/cuda/include/cuda_runtime_api.h:1683:101: error: use of enum ‘cudaDeviceP2PAttr’ without previous declaration
__ cudaError_t CUDARTAPI cudaDeviceGetP2PAttribute(int *value, enum cudaDeviceP
^
/usr/local/cuda/include/cuda_runtime_api.h:2920:102: error: use of enum ‘cudaFuncAttribute’ without previous declaration
_ cudaError_t CUDARTAPI cudaFuncSetAttribute(const void *func, enum cudaFuncAtt
^
In file included from /usr/local/cuda/include/channel_descriptor.h:62:0,
from /usr/local/cuda/include/cuda_runtime.h:90,
from /usr/include/cudnn.h:64,
from mnistCUDNN.cpp:30:
/usr/local/cuda/include/cuda_runtime_api.h:5763:92: error: use of enum ‘cudaMemoryAdvise’ without previous declaration
or_t CUDARTAPI cudaMemAdvise(const void *devPtr, size_t count, enum cudaMemoryA
^
/usr/local/cuda/include/cuda_runtime_api.h:5820:98: error: use of enum ‘cudaMemRangeAttribute’ without previous declaration
UDARTAPI cudaMemRangeGetAttribute(void *data, size_t dataSize, enum cudaMemRang
^
/usr/local/cuda/include/cuda_runtime_api.h:5857:102: error: use of enum ‘cudaMemRangeAttribute’ without previous declaration
TAPI cudaMemRangeGetAttributes(void **data, size_t *dataSizes, enum cudaMemRang
^
Makefile:200: recipe for target ‘mnistCUDNN.o’ failed

I managed to resolve it opening the file:
/usr/include/cudnn.h

…And changing the line:
#include “driver_types.h”

…to:
#include <driver_types.h>

This resolution was suggested in https://devtalk.nvidia.com/default/topic/1006726/cudnnv6-mnist-example-compile-errors/

Any suggestions for windows7, I cannot find this file cudnn.h, I explicitly added this driver_types.h, but still it doesnt work.

Hello everyone,

I have the same problem in win 10. I have Geforce 710M graphics card, I installed the cuda 9.2 for windows 10 and with cudaGetDeviceCount I got error number 35. Please help me what to do!

GeForce 710m is a Fermi device (compute capability 2.x)

Fermi devices are not supported by CUDA 9.x. The last CUDA version supporting those devices was CUDA 8.0.