cuda 8.0.61 samples on 375.66 driver: CUDA driver version is insufficient for CUDA runtime version

Hi,

on one of our Linux-Boxes (CentOS 7.1611, Quadro M4000, NVIDIA driver 375.66) I installed cuda 8.0.61 and compiled the samples. When I try to run one of the samples (e.g. “deviceQuery”) I invariably get the Output:

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

I am sure that half a year ago with cuda 8.0.44 and NVIDIA driver 375.26 the same program ran OK?

What did I miss?

your driver is not installed correctly, or else the driver that is installed is too old

you can figure out more by running

nvidia-smi

nvidia-smi
Fri Jul 14 13:38:16 2017
±--------------------- nvidia-smi
Fri Jul 14 13:38:16 2017
±----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro M4000 Off | 0000:02:00.0 On | N/A |
| 46% 32C P8 13W / 120W | 87MiB / 8120MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 7656 G /usr/bin/Xorg 76MiB |
| 0 7719 G gnome-shell 9MiB |
±----------------------------------------------------------------------------+
-------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro M4000 Off | 0000:02:00.0 On | N/A |
| 46% 32C P8 13W / 120W | 87MiB / 8120MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 7656 G /usr/bin/Xorg 76MiB |
| 0 7719 G gnome-shell 9MiB |
±----------------------------------------------------------------------------+

The driver installed is the latest I could get on the NVIDIA pages.

so your driver is not installed correctly
my guess would be a mixture of (previously) runfile install combined with package manager install, or vice versa, but there’s not enough info here to diagnose exactly.

My suggestion would be to follow the instructions in the linux install guide to clean out all old installs, then reinstall cuda 8.0.61. Before or after that, you can install 375.66, but make sure to use the same method as the CUDA install.

The first step may be to get familiar with the linux install guide.