Ailed to initialize NVML: Driver/library version mismatch

Hi

I’ve upgraded from kernel 4.5 to 4.11 on linux mint 20.04 (Ulyana), Ubuntu 20.04.
I also updated my nvidia-drivers after the update.
However nvidia-smi is not working and other nvidia tooling are putting out errors.

nvidia-smi Failed to initialize NVML: Driver/library version mismatch

nvidia-settings

ERROR: An internal driver error occurred
(nvidia-settings:53911): GLib-GObject-CRITICAL **: 14:04:51.644: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed
** (nvidia-settings:53911): CRITICAL **: 14:04:51.645: ctk_powermode_new: assertion ‘(ctrl_target != NULL) && (ctrl_target->h != NULL)’ failed
ERROR: nvidia-settings could not find the registry key file or the X server is not accessible. This file should have been installed along with this driver at /usr/share/nvidia/nvidia-application-profiles-key-documentation. The
application profiles will continue to work, but values cannot be prepopulated or validated, and will not be listed in the help text. Please see the README for possible values and descriptions.

ubuntu-drivers devices

WARNING:root:_pkg_get_support nvidia-driver-510-server: package has invalid Support PBheader, cannot determine support level
WARNING:root:_pkg_get_support nvidia-driver-510: package has invalid Support PBheader, cannot determine support level
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00001F15sv00001025sd00001400bc03sc00i00

Please post the output of
dpkg -l |grep nvidia
and
ls -l /usr/lib/x86_64-linux-gnu/libcuda*

Hi
Thank you for quick response. I also include the /proc/driver/nvidia/version which still points to 510 even though I have fallen back to 470 drivers.

dpkg -l | grep nvidia

ii libnvidia-cfg1-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-470 470.103.01-0ubuntu0.20.04.1 all Shared files used by the NVIDIA libraries
rc libnvidia-compute-460:amd64 470.103.01-0ubuntu0.20.04.1 amd64 Transitional package for libnvidia-compute-470
ii libnvidia-compute-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-compute-470:i386 470.103.01-0ubuntu0.20.04.1 i386 NVIDIA libcompute package
rc libnvidia-compute-510:amd64 510.54-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-decode-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-decode-470:i386 470.103.01-0ubuntu0.20.04.1 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-encode-470:i386 470.103.01-0ubuntu0.20.04.1 i386 NVENC Video Encoding runtime library
ii libnvidia-extra-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-fbc1-470:i386 470.103.01-0ubuntu0.20.04.1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-gl-470:i386 470.103.01-0ubuntu0.20.04.1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ifr1-470:amd64 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii libnvidia-ifr1-470:i386 470.103.01-0ubuntu0.20.04.1 i386 NVIDIA OpenGL-based Inband Frame Readback runtime library
rc linux-modules-nvidia-460-5.4.0-73-generic 5.4.0-73.82+1 amd64 Linux kernel nvidia modules for version 5.4.0-73
rc linux-modules-nvidia-470-5.4.0-107-generic 5.4.0-107.121 amd64 Linux kernel nvidia modules for version 5.4.0-107
rc linux-objects-nvidia-460-5.4.0-73-generic 5.4.0-73.82+1 amd64 Linux kernel nvidia modules for version 5.4.0-73 (objects)
rc linux-objects-nvidia-470-5.4.0-107-generic 5.4.0-107.121 amd64 Linux kernel nvidia modules for version 5.4.0-107 (objects)
ii nvidia-compute-utils-470 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA compute utilities
rc nvidia-compute-utils-510 510.54-0ubuntu0.20.04.1 amd64 NVIDIA compute utilities
ii nvidia-dkms-470 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA DKMS package
rc nvidia-dkms-510 510.54-0ubuntu0.20.04.1 amd64 NVIDIA DKMS package
ii nvidia-driver-470 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-470 470.103.01-0ubuntu0.20.04.1 amd64 Shared files used with the kernel module
rc nvidia-kernel-common-510 510.54-0ubuntu0.20.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-470 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA kernel source package
ii nvidia-prime 0.8.16~0.20.04.2 all Tools to enable NVIDIA’s Prime
ii nvidia-settings 470.57.01-0ubuntu0.20.04.3 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-470 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA driver support binaries
ii screen-resolution-extra 0.18build1 all Extension for the nvidia-settings control panel
ii xserver-xorg-video-nvidia-470 470.103.01-0ubuntu0.20.04.1 amd64 NVIDIA binary Xorg driver

ls -l /usr/lib/x86_64-linux-gnu/libcuda*

lrwxrwxrwx 1 root root 12 Jan 28 18:08 /usr/lib/x86_64-linux-gnu/libcuda.so → libcuda.so.1
lrwxrwxrwx 1 root root 21 Jan 28 18:08 /usr/lib/x86_64-linux-gnu/libcuda.so.1 → libcuda.so.470.103.01
-rw-r–r-- 1 root root 24216904 Jan 6 13:33 /usr/lib/x86_64-linux-gnu/libcuda.so.470.103.01

cat /proc/driver/nvidia/version

NVRM version: NVIDIA UNIX x86_64 Kernel Module 510.54 Tue Feb 8 04:42:21 UTC 2022
GCC version: gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

I have also run the nvidia-bug-report.sh
nvidia-bug-report.log.gz (132.1 KB)

The libraries are from 470 but the 510 kernel modules are still installed. Please run
sudo update-initramfs -u
and reboot to check if that resolves the issue.
Furthermore, please post the output of
dkms status
Also, please add kernel parameter
pci=noaer
because there are issues with your ethernet controller flooding the logs.

Ran the
update-initramfs -u

I’ve seen this printout couple of times.

update-initramfs: Generating /boot/initrd.img-5.11.0-46-generic I: The initramfs will attempt to resume from /dev/nvme0n1p6 I: (UUID=1df6fed3-2c1a-4a54-8299-4903cafcc751) I: Set the RESUME variable to override this.

Blockquote
sudo update-initramfs -u

nvidia-smi
Wed Apr 6 15:13:27 2022
±----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:01:00.0 Off | N/A |
| N/A 51C P8 5W / N/A | 498MiB / 5934MiB | 7% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 950 G /usr/lib/xorg/Xorg 193MiB |
| 0 N/A N/A 2356 G cinnamon 83MiB |
| 0 N/A N/A 2608 G …AAAAAAAAA= --shared-files 38MiB |
| 0 N/A N/A 2803 G /usr/lib/firefox/firefox 135MiB |
| 0 N/A N/A 3480 G …932859119650238057,131072 43MiB |
±----------------------------------------------------------------------------+

This solved the issue. Now 470 is working.
Thankx alot for your help.
br
Thomas

Everything is working now, but I cannot seem to get rid of this warning:

ubuntu-drivers devices

WARNING:root:_pkg_get_support nvidia-driver-510: package has invalid Support PBheader, cannot determine support level
WARNING:root:_pkg_get_support nvidia-driver-510-server: package has invalid Support PBheader, cannot determine support level
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00001F15sv00001025sd00001400bc03sc00i00

That’s some ubuntu packaging issue, just ignore or report to canonical.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.