Nvidia-smi: No devices were found

I updated the nvidia driver from 470 to 525 and cuda toolkit.

cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 525.78.01 Release Build (dvs-builder@U16-J11-10-2) Mon Dec 26 05:50:21 UTC 2022
GCC version: gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0

For some reason, nvidia-smi doesn’t recognize GPUs, which was working fine before the driver update:
nvidia-smi
No devices were found

And, deviceQuery fails:
./deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 100
→ no CUDA-capable device is detected
Result = FAIL

But lspci gives the following output:
lspci -nn | grep ‘[03’
67:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] [10de:1e30] (rev a1)
68:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] [10de:1e30] (rev a1)

So, I am am not sure what could be the issue. Any suggestions/pointers would be greatly helpful!

Thank you!

Answering my own question. It turns out, installing the non “open” version of the driver solves the problem, even though the open version is the one recommended by the “ubuntu-drivers devices” command. i.e., I installed “nvidia-driver-525” instead of “nvidia-driver-525-open” and it worked. I did this by installing “cuda-drivers-525”, which seems to be a meta package for the nvidia driver. Hope it helps others.