Nvidia-smi: No devices were found

Graphics / Linux Linux
1

I updated the nvidia driver from 470 to 525 and cuda toolkit.

cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 525.78.01 Release Build (dvs-builder@U16-J11-10-2) Mon Dec 26 05:50:21 UTC 2022
GCC version: gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0

For some reason, nvidia-smi doesn’t recognize GPUs, which was working fine before the driver update:
nvidia-smi
No devices were found

And, deviceQuery fails:
./deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 100
→ no CUDA-capable device is detected
Result = FAIL

But lspci gives the following output:
lspci -nn | grep ‘[03’
67:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] [10de:1e30] (rev a1)
68:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] [10de:1e30] (rev a1)

So, I am am not sure what could be the issue. Any suggestions/pointers would be greatly helpful!

Thank you!

1 Like
2

Answering my own question. It turns out, installing the non “open” version of the driver solves the problem, even though the open version is the one recommended by the “ubuntu-drivers devices” command. i.e., I installed “nvidia-driver-525” instead of “nvidia-driver-525-open” and it worked. I did this by installing “cuda-drivers-525”, which seems to be a meta package for the nvidia driver. Hope it helps others.

3 Likes