Failed to initialize NVML: Driver/library version mismatch

Graphics / Linux Linux
, , , , , ,
1

Hi all. I have a system running on Ubuntu.

Ubuntu 22.04.2 LTS
Kernel version: 5.19.0-051900-generic
CPU: Intel Xeon Gold 6448Y
GPU: NVIDIA Corporation GA102GL [A10]

And I’m getting this error when I’m trying to use nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

Tried looking everywhere for a solution for this. I just want to figure out if the card is being utilised when I am running my workloads.

(See bug report attached)
nvidia-bug-report.log.gz (96.5 KB)

This is what I am seeing in the bug report.

NVRM: API mismatch: the client has the version 525.105.17, but
NVRM: this kernel module has the version 525.89.02. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.

2 Likes
2

This helped me cuda - Nvidia NVML Driver/library version mismatch - Stack Overflow

1 Like
3

Had the same issue on my Ubuntu 22.04.4 LTS laptop!

Do the following steps:

dpkg -l | grep nvidia not required but just to check what versions are installed
sudo apt --purge remove "*nvidia*"
sudo ubuntu-drivers devices View all hardware NVidia devices which need drivers, and which packages

And then install the default and latest available on your machine:
sudo ubuntu-drivers install
sudo reboot would be also helpful (but optional)

$ nvidia-smi

Fri Apr  5 14:21:44 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A1000 6GB Lap...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   57C    P0             312W /  35W |      8MiB /  6144MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3071      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+