Sample devieQuery cuda program error in Cuda 10.0 and Centos 7

Accelerated Computing CUDA CUDA Setup and Installation
1

I have a Intel Xeon machine with NVIDIA GeForce1080 GTX configured and CentOS 7 as operating system. I have installed NVIDIA-driver 410.48 and cuda-toolkit 10.0. After compiling the cuda-samples, i tried to run ./deviceQuery.
But it throws like this

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 30
-> unknown error
Result = FAIL


lspci | grep VGA

01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)


nvidia-smi

Tue Feb 12 11:06:22 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   56C    P0    46W / 240W |    201MiB /  8119MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     20963      G   /usr/bin/X                                    96MiB |
|    0     21546      G   /usr/bin/gnome-shell                         103MiB |
+-----------------------------------------------------------------------------+

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.13

PATH & LD_LIBRARY_PATH

PATH =/usr/local/cuda-10.0/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/local/sbin:
LD_LIBRARY_PATH = /usr/local/cuda-10.0/lib64:/usr/local/cuda/lib64:


lsmod | grep nvidia

nvidia_drm             39819  3 
nvidia_modeset       1048491  6 nvidia_drm
nvidia              16805034  276 nvidia_modeset
drm_kms_helper        179394  1 nvidia_drm
drm                   429744  6 drm_kms_helper,nvidia_drm
ipmi_msghandler        56032  2 ipmi_devintf,nvidia

lsmod | grep nvidia-uvm
no output

dmesg | grep NVRM

[    8.237489] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  410.93  Thu Dec 20 17:01:16 CST 2018 (using threaded interrupts)

is this problem anything related to modprobe or nvidia-uvm?
Thanking in advance.

2

I encountered similar issue as you. I tested on CentOS and RH7 machines, each has 2 to 4 GPUs installed. The command nvidia-smi can properly show all GPUs installed on the system, however a simple deviceQuery program failed to detect any GPU…Another Ubuntu 18.04 machine doesn’t have this problem.

Here is a toy code that queries CUDA devices and it output nothing.

I am using CUDA-10 and gcc-7.

#include <cuda_runtime.h>
#include <stdio.h>

int main() {
  int nDevices;

  cudaGetDeviceCount(&nDevices);
  for (int i = 0; i < nDevices; i++) {
    cudaDeviceProp prop;
    cudaGetDeviceProperties(&prop, i);
    printf("Device Number: %d\n", i);
    printf("  Device name: %s\n", prop.name);
    printf("  Memory Clock Rate (KHz): %d\n", prop.memoryClockRate);
    printf("  Memory Bus Width (bits): %d\n", prop.memoryBusWidth);
    printf("  Peak Memory Bandwidth (GB/s): %f\n\n",
           2.0 * prop.memoryClockRate * (prop.memoryBusWidth / 8) / 1.0e6);
  }
}
3

Please reply the output of these commands.

nvidia-smi

nvcc --version

I resolved this problem by auto-removing (purge in ubuntu) all the NVIDIA-driver and cuda driver. And installed nvidia-driver and cuda toolkit using the cuda run file.

PS: cuda installation run file have a specific nvidia-driver version use that.