deviceQuery not working, nvidia-smi working

I can not figure out why deviceQuery doesn’t show the information about the devices

$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL
$ pwd
/storage/users/mahmood/NVIDIA_CUDA-10.0_Samples/1_Utilities/deviceQuery
$ nvidia-smi
Thu Sep  5 20:41:18 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:02:00.0 Off |                  N/A |
| 29%   55C    P2    85W / 250W |  10723MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 980 Ti  Off  | 00000000:81:00.0 Off |                  N/A |
| 93%   87C    P2   188W / 250W |   5970MiB /  6083MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
$ ls /usr/local/cuda-10.0/lib64/libcudart*
/usr/local/cuda-10.0/lib64/libcudart.so       /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
/usr/local/cuda-10.0/lib64/libcudart.so.10.0  /usr/local/cuda-10.0/lib64/libcudart_static.a

I also even tried with a user installation of toolkit (without display driver) and got the same error.

Any idea?

Do you have CUDA_VISIBLE_DEVICES set to an empty string?

It was 7 as I login to the node. I don’t know why administrator have set that!
Setting to 1 fixed the problem. Thanks.