Nvidia-smi failed to initialize NVML (driver/library version mismatch)

I am running Ubuntu 20.04 and I am often facing this nvidia driver issue. I have the recommended proprietary nvidia driver installed. Reboot did not work. Logs are attached.
nvidia-bug-report.log.gz (168.2 KB)

[  408.222979] NVRM: API mismatch: the client has the version 515.65.01, but
               NVRM: this kernel module has the version 515.48.07.  Please
               NVRM: make sure that this kernel module and all NVIDIA driver
               NVRM: components have the same version.

Please post the output of
dkms status

This just popped up for me as well. Two different machines one Ubuntu 20.04 the other 22.04.

❯ dkms status
nvidia, 470.141.03, 5.15.0-41-generic, x86_64: installed
nvidia, 470.141.03, 5.15.0-43-generic, x86_64: installed

Hi, the same is happening to me with the same version mismatch (515.65.01 and 515.48.07) and like @mbasuth1 this isn’t the first time it has happened, and rebooting did not fix the issue. Only difference is that I get no output when running dkms status. Any help would be greatly appreciated!

@mbasuth1 @eloop Maybe you’ve tried this already, but I just did sudo apt update and upgrade (there was an error for that but it suggested doing sudo apt --fix-broken install which I did, after which I could upgrade with no issues) and then did a reboot, and tada! It all works again 🙏🏽 Hopefully this can help you!

@generix
dkms status output is:
nvidia, 515.65.01, 5.15.0-43-generic, x86_64: installed

do i need to uninstall the current nvidia drivers and reinstall them with 515.48.07 version?

@leozinho2r
unfortunately, the update and upgrade did not fix my problem :(

@mbasuth1 Looks like everything is correctly installed, just the initrd still holds the old driver. Please run
sudo update-initramfs -u
and reboot.
Then check if the correct kernel modules load.

@generix
I am getting this error when I tried to update initramfs. Would you know how I could fix this?

administrator@administrator-Nuvo-8108GC-Series:~$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic
Error 24 : Write error : cannot write compressed block
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.15.0-43-generic with 1.
administrator@administrator-Nuvo-8108GC-Series:~$

Might be an extra /boot or efi partition that’s full. Please post the output of
df

@generix
Sure, attached in the txt file>>
df.txt (4 KB)

bpool/BOOT/ubuntu_1kxay8 354304 310912 43392 88% /boot
Please uninstall old kernel images, then try again.

@generix I am trying to remove the old kernel images but it seems like it is failing because of initramfs? How can I resolve this? I tried `sudo apt autoremove --purge too and had the same error as shown below.

administrator@administrator-Nuvo-8108GC-Series:~$ dpkg --list | grep linux-image
rc  linux-image-5.11.0-27-generic               5.11.0-27.29~20.04.1                amd64        Signed kernel image generic
rc  linux-image-5.13.0-41-generic               5.13.0-41.46~20.04.1                amd64        Signed kernel image generic
rc  linux-image-5.13.0-44-generic               5.13.0-44.49~20.04.1                amd64        Signed kernel image generic
ii  linux-image-5.15.0-41-generic               5.15.0-41.44~20.04.1                amd64        Signed kernel image generic
ii  linux-image-5.15.0-43-generic               5.15.0-43.46~20.04.1                amd64        Signed kernel image generic
ii  linux-image-generic-hwe-20.04               5.15.0.43.46~20.04.14               amd64        Generic Linux kernel image

administrator@administrator-Nuvo-8108GC-Series:~$ uname -r
5.15.0-43-generic

administrator@administrator-Nuvo-8108GC-Series:~$ sudo apt-get purge linux-image-5.11.0-27-generic
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-image-5.11.0-27-generic*
0 upgraded, 0 newly installed, 1 to remove and 17 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Requesting to save current system state
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 9%.
Please remove some states manually to free up space. 
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.15.0-43-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.15.0-43-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-firmware
 initramfs-tools
ZSys is adding automatic system snapshot to GRUB menu
E: Sub-process /usr/bin/dpkg returned an error code (1)

administrator@administrator-Nuvo-8108GC-Series:~$ df -h /boot
Filesystem                Size  Used Avail Use% Mounted on
bpool/BOOT/ubuntu_1kxay8  346M  304M   43M  88% /boot

administrator@administrator-Nuvo-8108GC-Series:~$ sudo apt purge linux*5.11.0-27*
[sudo] password for administrator: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'linux-modules-nvidia-450-server-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-450-server-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-460-server-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-image-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-image-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-cloud-tools-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-450-server-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-418-server-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-470-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-390-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-418-server-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-buildinfo-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-460-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-460-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-390-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-418-server-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-hwe-5.11-cloud-tools-5.11.0-27' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-390-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-tools-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-418-server-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-image-unsigned-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-headers-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-extra-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-460-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-extra-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-470-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-image-unsigned-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-headers-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-470-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-buildinfo-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-cloud-tools-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-hwe-5.11-tools-5.11.0-27' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-hwe-5.11-headers-5.11.0-27' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-signatures-nvidia-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-460-server-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-tools-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-450-server-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-470-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-signatures-nvidia-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-460-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-objects-nvidia-390-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-460-server-5.11.0-27-generic' for glob 'linux*5.11.0-27*'
Note, selecting 'linux-modules-nvidia-460-server-5.11.0-27-lowlatency' for glob 'linux*5.11.0-27*'
Package 'linux-modules-extra-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-buildinfo-5.11.0-27-generic' is not installed, so not removed
Package 'linux-buildinfo-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-cloud-tools-5.11.0-27-generic' is not installed, so not removed
Package 'linux-cloud-tools-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-headers-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-hwe-5.11-cloud-tools-5.11.0-27' is not installed, so not removed
Package 'linux-hwe-5.11-tools-5.11.0-27' is not installed, so not removed
Package 'linux-image-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-image-unsigned-5.11.0-27-generic' is not installed, so not removed
Package 'linux-image-unsigned-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-tools-5.11.0-27-generic' is not installed, so not removed
Package 'linux-tools-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-nvidia-390-5.11.0-27-generic' is not installed, so not removed
Package 'linux-modules-nvidia-390-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-nvidia-418-server-5.11.0-27-generic' is not installed, so not removed
Package 'linux-modules-nvidia-418-server-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-nvidia-450-server-5.11.0-27-generic' is not installed, so not removed
Package 'linux-modules-nvidia-450-server-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-nvidia-460-5.11.0-27-generic' is not installed, so not removed
Package 'linux-modules-nvidia-460-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-nvidia-460-server-5.11.0-27-generic' is not installed, so not removed
Package 'linux-modules-nvidia-460-server-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-modules-nvidia-470-5.11.0-27-generic' is not installed, so not removed
Package 'linux-modules-nvidia-470-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-objects-nvidia-390-5.11.0-27-generic' is not installed, so not removed
Package 'linux-objects-nvidia-390-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-objects-nvidia-418-server-5.11.0-27-generic' is not installed, so not removed
Package 'linux-objects-nvidia-418-server-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-objects-nvidia-450-server-5.11.0-27-generic' is not installed, so not removed
Package 'linux-objects-nvidia-450-server-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-objects-nvidia-460-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-objects-nvidia-460-server-5.11.0-27-generic' is not installed, so not removed
Package 'linux-objects-nvidia-460-server-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-objects-nvidia-470-5.11.0-27-generic' is not installed, so not removed
Package 'linux-objects-nvidia-470-5.11.0-27-lowlatency' is not installed, so not removed
Package 'linux-signatures-nvidia-5.11.0-27-generic' is not installed, so not removed
Package 'linux-signatures-nvidia-5.11.0-27-lowlatency' is not installed, so not removed
The following packages will be REMOVED:
  linux-headers-5.11.0-27-generic* linux-hwe-5.11-headers-5.11.0-27* linux-image-5.11.0-27-generic*
  linux-modules-5.11.0-27-generic* linux-modules-extra-5.11.0-27-generic*
  linux-objects-nvidia-460-5.11.0-27-generic*
0 upgraded, 0 newly installed, 6 to remove and 17 not upgraded.
2 not fully installed or removed.
After this operation, 90.3 MB disk space will be freed.
Do you want to continue? [Y/n] Y
Requesting to save current system state
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 9%.
Please remove some states manually to free up space. 
(Reading database ... 334852 files and directories currently installed.)
Removing linux-headers-5.11.0-27-generic (5.11.0-27.29~20.04.1) ...
dpkg: warning: while removing linux-headers-5.11.0-27-generic, directory '/lib/modules/5.11.0-27-generic' not empty so not removed
Removing linux-hwe-5.11-headers-5.11.0-27 (5.11.0-27.29~20.04.1) ...
Setting up initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.15.0-43-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-firmware
ZSys is adding automatic system snapshot to GRUB menu
E: Sub-process /usr/bin/dpkg returned an error code (1)

administrator@administrator-Nuvo-8108GC-Series:~$ 

Check the directory’s content, delete manually, then remove the packages. Having an extra boot partition only 400MB small isn’t really useful when having the nvidia driver (>50MB).