Nvidia-smi results in Failed to initialize NVML: Driver/library version mismatch (Ubuntu 20.04.6 LTS)

I have a problem on Ubuntu 20.04 with nvidia drivers.

Everything is fine when the system starts up. However, after a few days, nvidia-smi return: Failed to initialize NVML: Driver/library version mismatch.

lspci -v | grep VGA
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52) (prog-if 00 [VGA controller])
31:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1) (prog-if 00 [VGA controller])
4b:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1) (prog-if 00 [VGA controller])
b1:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1) (prog-if 00 [VGA controller])
ca:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1) (prog-if 00 [VGA controller])

dpkg -l |grep nvidia:
ii libnvidia-cfg1-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-525 525.105.17-0ubuntu0.20.04.1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-compute-525:i386 525.105.17-0ubuntu0.20.04.1 i386 NVIDIA libcompute package
ii libnvidia-decode-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-decode-525:i386 525.105.17-0ubuntu0.20.04.1 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-encode-525:i386 525.105.17-0ubuntu0.20.04.1 i386 NVENC Video Encoding runtime library
ii libnvidia-extra-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-fbc1-525:i386 525.105.17-0ubuntu0.20.04.1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-525:amd64 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-gl-525:i386 525.105.17-0ubuntu0.20.04.1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii nvidia-compute-utils-525 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA compute utilities
ii nvidia-dkms-525 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA DKMS package
ii nvidia-driver-525 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-525 525.105.17-0ubuntu0.20.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-525 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA kernel source package
ii nvidia-prime 0.8.16~0.20.04.2 all Tools to enable NVIDIA’s Prime
ii nvidia-settings 470.57.01-0ubuntu0.20.04.3 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-525 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA driver support binaries
ii screen-resolution-extra 0.18build1 all Extension for the nvidia-settings control panel
ii xserver-xorg-video-nvidia-525 525.105.17-0ubuntu0.20.04.1 amd64 NVIDIA binary Xorg driver

ls -l /usr/lib/x86_64-linux-gnu/nvidia:
lrwxrwxrwx 1 root root 27 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.0 → libEGL_nvidia.so.525.105.17
-rw-r–r-- 1 root root 1333232 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.525.105.17
lrwxrwxrwx 1 root root 33 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.1 → libGLESv1_CM_nvidia.so.525.105.17
-rw-r–r-- 1 root root 67880 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.525.105.17
lrwxrwxrwx 1 root root 30 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.2 → libGLESv2_nvidia.so.525.105.17
-rw-r–r-- 1 root root 117032 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.525.105.17
lrwxrwxrwx 1 root root 27 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0 → libGLX_nvidia.so.525.105.17
-rw-r–r-- 1 root root 1191352 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.525.105.17
lrwxrwxrwx 1 root root 24 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so → libnvidia-allocator.so.1
lrwxrwxrwx 1 root root 33 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.1 → libnvidia-allocator.so.525.105.17
-rw-r–r-- 1 root root 156216 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.525.105.17
lrwxrwxrwx 1 root root 18 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so → libnvidia-cfg.so.1
lrwxrwxrwx 1 root root 27 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.1 → libnvidia-cfg.so.525.105.17
-rw-r–r-- 1 root root 262512 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.525.105.17
-rw-r–r-- 1 root root 56309912 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.525.105.17
-rw-r–r-- 1 root root 42054840 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.525.105.17
lrwxrwxrwx 1 root root 26 3月 29 11:09 /usr/lib/x86_64-linux-gnu/libnvidia-egl-gbm.so.1 → libnvidia-egl-gbm.so.1.1.0
-rw-r–r-- 1 root root 22656 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-egl-gbm.so.1.1.0
lrwxrwxrwx 1 root root 21 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so → libnvidia-encode.so.1
lrwxrwxrwx 1 root root 30 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1 → libnvidia-encode.so.525.105.17
-rw-r–r-- 1 root root 198720 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.525.105.17
lrwxrwxrwx 1 root root 18 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so → libnvidia-fbc.so.1
lrwxrwxrwx 1 root root 27 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.1 → libnvidia-fbc.so.525.105.17
-rw-r–r-- 1 root root 133816 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.525.105.17
-rw-r–r-- 1 root root 44282616 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.525.105.17
-rw-r–r-- 1 root root 652264 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.525.105.17
-rw-r–r-- 1 root root 25056984 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.525.105.17
lrwxrwxrwx 1 root root 17 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so → libnvidia-ml.so.1
lrwxrwxrwx 1 root root 26 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 → libnvidia-ml.so.525.105.17
-rw-r–r-- 1 root root 1806968 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.525.105.17
lrwxrwxrwx 1 root root 27 4月 9 06:40 /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.1 → libnvidia-ngx.so.525.105.17
-rw-r–r-- 1 root root 3310216 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.525.105.17
lrwxrwxrwx 1 root root 19 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so → libnvidia-nvvm.so.4
lrwxrwxrwx 1 root root 28 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.4 → libnvidia-nvvm.so.525.105.17
-rw-r–r-- 1 root root 85957696 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.525.105.17
lrwxrwxrwx 1 root root 30 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1 → libnvidia-opencl.so.525.105.17
-rw-r–r-- 1 root root 22991312 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.525.105.17
lrwxrwxrwx 1 root root 26 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so → libnvidia-opticalflow.so.1
lrwxrwxrwx 1 root root 35 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1 → libnvidia-opticalflow.so.525.105.17
-rw-r–r-- 1 root root 67584 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.525.105.17
lrwxrwxrwx 1 root root 29 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so → libnvidia-ptxjitcompiler.so.1
lrwxrwxrwx 1 root root 38 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1 → libnvidia-ptxjitcompiler.so.525.105.17
-rw-r–r-- 1 root root 20727192 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.525.105.17
-rw-r–r-- 1 root root 118080512 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.525.105.17
-rw-r–r-- 1 root root 18528 3月 27 19:40 /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.525.105.17

nvidia-bug-report.log.gz (222.0 KB)

what should I do? thanks!

Your driver got updated but the old kernel modules are still loaded. Please reboot.

dpkg -l|grep nvidia
ii libnvidia-cfg1-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-525 525.116.04-0ubuntu0.20.04.1 all Shared files used by the NVIDIA libraries
rc libnvidia-compute-515:amd64 515.86.01-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-compute-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-compute-525:i386 525.116.04-0ubuntu0.20.04.1 i386 NVIDIA libcompute package
ii libnvidia-container-tools 1.12.1-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.12.1-1 amd64 NVIDIA container runtime library
ii libnvidia-decode-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-decode-525:i386 525.116.04-0ubuntu0.20.04.1 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-encode-525:i386 525.116.04-0ubuntu0.20.04.1 i386 NVENC Video Encoding runtime library
ii libnvidia-extra-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-fbc1-525:i386 525.116.04-0ubuntu0.20.04.1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-525:amd64 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-gl-525:i386 525.116.04-0ubuntu0.20.04.1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ml-dev 10.1.243-3 amd64 NVIDIA Management Library (NVML) development files
rc linux-modules-nvidia-515-5.15.0-46-generic 5.15.0-46.49~20.04.1+1 amd64 Linux kernel nvidia modules for version 5.15.0-46
rc linux-modules-nvidia-515-5.15.0-60-generic 5.15.0-60.66~20.04.1 amd64 Linux kernel nvidia modules for version 5.15.0-60
rc linux-objects-nvidia-515-5.15.0-46-generic 5.15.0-46.49~20.04.1+1 amd64 Linux kernel nvidia modules for version 5.15.0-46 (objects)
rc linux-objects-nvidia-515-5.15.0-60-generic 5.15.0-60.66~20.04.1 amd64 Linux kernel nvidia modules for version 5.15.0-60 (objects)
rc nvidia-compute-utils-515 515.86.01-0ubuntu0.20.04.1 amd64 NVIDIA compute utilities
ii nvidia-compute-utils-525 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA compute utilities
ii nvidia-container-toolkit 1.12.1-1 amd64 NVIDIA Container toolkit
ii nvidia-container-toolkit-base 1.12.1-1 amd64 NVIDIA Container Toolkit Base
ii nvidia-cuda-dev 10.1.243-3 amd64 NVIDIA CUDA development files
ii nvidia-cuda-doc 10.1.243-3 all NVIDIA CUDA and OpenCL documentation
ii nvidia-cuda-gdb 10.1.243-3 amd64 NVIDIA CUDA Debugger (GDB)
ii nvidia-cuda-toolkit 10.1.243-3 amd64 NVIDIA CUDA development toolkit
ii nvidia-dkms-525 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA DKMS package
ii nvidia-driver-525 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA driver metapackage
rc nvidia-kernel-common-515 515.86.01-0ubuntu0.20.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-common-525 525.116.04-0ubuntu0.20.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-525 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA kernel source package
ii nvidia-opencl-dev:amd64 10.1.243-3 amd64 NVIDIA OpenCL development files
ii nvidia-prime 0.8.16~0.20.04.2 all Tools to enable NVIDIA’s Prime
ii nvidia-profiler 10.1.243-3 amd64 NVIDIA Profiler for CUDA and OpenCL
ii nvidia-settings 530.30.02-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-525 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA driver support binaries
ii nvidia-visual-profiler 10.1.243-3 amd64 NVIDIA Visual Profiler for CUDA and OpenCL
ii screen-resolution-extra 0.18build1 all Extension for the nvidia-settings control panel
ii xserver-xorg-video-nvidia-525 525.116.04-0ubuntu0.20.04.1 amd64 NVIDIA binary Xorg driver

nvidia-bug-report.log.gz (174.1 KB)

Please help me