Failed to initialize NVML: Driver/library version mismatch

Hi all. I have a system running on Ubuntu.

Ubuntu 22.04.2 LTS
Kernel version: 5.19.0-051900-generic
CPU: Intel Xeon Gold 6448Y
GPU: NVIDIA Corporation GA102GL [A10]

And I’m getting this error when I’m trying to use nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

Tried looking everywhere for a solution for this. I just want to figure out if the card is being utilised when I am running my workloads.

(See bug report attached)
nvidia-bug-report.log.gz (96.5 KB)

This is what I am seeing in the bug report.

NVRM: API mismatch: the client has the version 525.105.17, but
NVRM: this kernel module has the version 525.89.02. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.

1 Like

This helped me cuda - Nvidia NVML Driver/library version mismatch - Stack Overflow

1 Like