Hi,
After a power outage and unexpected shutdown of the system, I now see that nvidia-smi works while deviceQuery fails to work. I remember a similar problem which was fixed by setting CUDA_VISIBLE_DEVICES variable. I don’t know but maybe a lock file exists which prevent deviceQuery to work.
I see limited information about error 999 aad it is not clear what is the real issue.
mahmood@threadripper:~/NVIDIA_CUDA-10.1_Samples/1_Utilities/deviceQuery$ make
/usr/local/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o deviceQuery.o -c deviceQuery.cpp
/usr/local/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o deviceQuery deviceQuery.o
mkdir -p ../../bin/x86_64/linux/release
cp deviceQuery ../../bin/x86_64/linux/release
mahmood@threadripper:~/NVIDIA_CUDA-10.1_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 999
-> unknown error
Result = FAIL
mahmood@threadripper:~/NVIDIA_CUDA-10.1_Samples/1_Utilities/deviceQuery$ nvidia-smi
Sun Oct 6 11:57:36 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:41:00.0 On | N/A |
| 31% 39C P8 7W / 260W | 211MiB / 10984MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1141 G /usr/lib/xorg/Xorg 16MiB |
| 0 1170 G /usr/bin/gnome-shell 44MiB |
| 0 2235 G /usr/lib/xorg/Xorg 60MiB |
| 0 2404 G /usr/bin/gnome-shell 78MiB |
| 0 7072 G /opt/firefox/firefox-bin 6MiB |
+-----------------------------------------------------------------------------+
mahmood@threadripper:~/NVIDIA_CUDA-10.1_Samples/1_Utilities/deviceQuery$ echo $CUDA_VISIBLE_DEVICES
0