Failed to initialize NVML: Driver/library version mismatch

Graphics / Linux Linux
1

I am running Ubuntu 20.04 and have two GPUs a 970 and 3060 installed.

NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2

I am running some custom software - that uses CUDA. It now shows an error :[ERR] Failed to allocate device memory (cuda error: forward compatibility was attempted on non supported HW)

When I run nvidia-smi I get Error: nvmlInit(): Driver/library version mismatch

2

nvidia-bug-report.log.gz (136.8 KB)

3

Seems like you’re running an old kernel, 5.13.0-28, while the current kernel is 5.13.0-30 and meanwhile, driver v470 was installed so you now have a mismatch of library versions. Furthermore, you seem to have installed the full cuda package including driver over the repo driver. Rather boot the correct kernel, purge anything cuda/nvidia, install the driver from repo using Software&Updates, the install just the cuda-toolkit, e.g.
sudo apt install cuda-toolkit-11-2