# Failed to initialize NVML: Driver/library version mismatch on RHEL8

**URL:** <https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355>\
**Category:** Linux\
**Created:** [March 14, 2022, 2:13pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355 "2022-03-14T14:13:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![dade.james](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dade.james](https://forums.developer.nvidia.com/u/dade.james)\
**Post date:** [March 14, 2022, 2:13pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/1 "2022-03-14T14:13:57Z")

</div>

Getting this error when trying to run nvidia-smi on a RHEL 8.5 server

NVRM version: NVIDIA UNIX x86\_64 Kernel Module 510.47.03 is my current driver.

01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eH3 (rev 02) (prog-if 00 [VGA controller])  
Subsystem: Hewlett Packard Enterprise iLO5 VGA  
8e:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])  
9c:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])  
ce:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])  
dc:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])

---

<div class="post-metadata">

**Author:** ![dade.james](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dade.james](https://forums.developer.nvidia.com/u/dade.james)\
**Post date:** [March 14, 2022, 2:21pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/2 "2022-03-14T14:21:46Z")

</div>

[root@osmr7-gpuserver sa]# rpm -qa | grep -i nvidia  
nvidia-persistenced-510.47.03-1.el8.x86\_64  
nvidia-driver-devel-510.47.03-1.el8.x86\_64  
nvidia-xconfig-510.47.03-1.el8.x86\_64  
nvidia-driver-cuda-libs-510.47.03-1.el8.x86\_64  
nvidia-driver-510.47.03-1.el8.x86\_64  
nvidia-modprobe-510.47.03-1.el8.x86\_64  
nvidia-driver-NVML-510.47.03-1.el8.x86\_64  
nvidia-libXNVCtrl-devel-510.47.03-1.el8.x86\_64  
dnf-plugin-nvidia-2.0-1.el8.noarch  
nvidia-libXNVCtrl-510.47.03-1.el8.x86\_64  
nvidia-driver-NvFBCOpenGL-510.47.03-1.el8.x86\_64  
nvidia-kmod-common-510.47.03-1.el8.noarch  
nvidia-settings-510.47.03-1.el8.x86\_64  
nvidia-driver-libs-510.47.03-1.el8.x86\_64  
kmod-nvidia-latest-dkms-510.47.03-1.el8.x86\_64  
nvidia-driver-cuda-510.47.03-1.el8.x86\_64

---

<div class="post-metadata">

**Author:** ![generix](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/generix/32/16024_2.png) [@generix](https://forums.developer.nvidia.com/u/generix)\
**Post date:** [March 14, 2022, 2:26pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/3 "2022-03-14T14:26:46Z")

</div>

Please check for leftovers from an earlier driver install,  
ls -l /usr/lib64/libcuda\*

---

<div class="post-metadata">

**Author:** ![dade.james](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dade.james](https://forums.developer.nvidia.com/u/dade.james)\
**Post date:** [March 14, 2022, 2:34pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/4 "2022-03-14T14:34:37Z")

</div>

(base) [root@osmr7-gpuserver sa]# ls -l /usr/lib64/libcuda\*  
lrwxrwxrwx. 1 root root 12 Feb 17 10:01 /usr/lib64/libcuda.so → libcuda.so.1  
lrwxrwxrwx. 1 root root 20 Feb 17 10:01 /usr/lib64/libcuda.so.1 → libcuda.so.470.57.02  
-rwxr-xr-x. 1 root root 22267208 Feb 17 10:01 /usr/lib64/libcuda.so.470.57.02

(base) [root@osmr7-gpuserver sa]# rpm -qa | grep -i cuda  
cuda-nvvp-11-6-11.6.112-1.x86\_64  
cuda-memcheck-11-6-11.6.112-1.x86\_64  
cuda-cuxxfilt-11-6-11.6.112-1.x86\_64  
cuda-compiler-11-6-11.6.1-1.x86\_64  
cuda-nvml-devel-11-6-11.6.55-1.x86\_64  
nvidia-driver-cuda-libs-510.47.03-1.el8.x86\_64  
cuda-libraries-devel-11-6-11.6.1-1.x86\_64  
cuda-visual-tools-11-6-11.6.1-1.x86\_64  
cuda-toolkit-11-6-config-common-11.6.55-1.noarch  
cuda-cudart-devel-11-6-11.6.55-1.x86\_64  
cuda-driver-devel-11-6-11.6.55-1.x86\_64  
cuda-nvrtc-11-6-11.6.112-1.x86\_64  
cuda-nvprune-11-6-11.6.112-1.x86\_64  
cuda-cupti-11-6-11.6.112-1.x86\_64  
cuda-tools-11-6-11.6.1-1.x86\_64  
cuda-repo-rhel8-11-6-local-11.6.0\_510.39.01-1.x86\_64  
cuda-nvdisasm-11-6-11.6.104-1.x86\_64  
cuda-nsight-11-6-11.6.112-1.x86\_64  
cuda-11.6.1-1.x86\_64  
cuda-toolkit-config-common-11.6.55-1.noarch  
cuda-nvrtc-devel-11-6-11.6.112-1.x86\_64  
cuda-documentation-11-6-11.6.112-1.x86\_64  
cuda-sanitizer-11-6-11.6.112-1.x86\_64  
cuda-cudart-11-6-11.6.55-1.x86\_64  
cuda-samples-11-6-11.6.101-1.x86\_64  
cuda-libraries-11-6-11.6.1-1.x86\_64  
cuda-nvcc-11-6-11.6.112-1.x86\_64  
cuda-cuobjdump-11-6-11.6.112-1.x86\_64  
cuda-toolkit-11-6-11.6.1-1.x86\_64  
cuda-demo-suite-11-6-11.6.55-1.x86\_64  
cuda-drivers-510.47.03-1.x86\_64  
cuda-nsight-compute-11-6-11.6.1-1.x86\_64  
cuda-toolkit-11-config-common-11.6.55-1.noarch  
cuda-cccl-11-6-11.6.55-1.x86\_64  
cuda-nvtx-11-6-11.6.112-1.x86\_64  
cuda-command-line-tools-11-6-11.6.1-1.x86\_64  
cuda-nvprof-11-6-11.6.112-1.x86\_64  
cuda-nsight-systems-11-6-11.6.1-1.x86\_64  
cuda-11-6-11.6.1-1.x86\_64  
nvidia-driver-cuda-510.47.03-1.el8.x86\_64  
cuda-runtime-11-6-11.6.1-1.x86\_64  
cuda-gdb-11-6-11.6.112-1.x86\_64  
(base) [root@osmr7-gpuserver sa]#

---

<div class="post-metadata">

**Author:** ![generix](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/generix/32/16024_2.png) [@generix](https://forums.developer.nvidia.com/u/generix)\
**Post date:** [March 16, 2022, 5:31pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/5 "2022-03-16T17:31:35Z")

</div>

Looks like you have leftovers from an ealier 470 driver install. Did you previously use the runfile installer? Please also check for more leftovers  
`ls -l /usr/lib64/*nvidia*`

---

<div class="post-metadata">

**Author:** ![dade.james](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dade.james](https://forums.developer.nvidia.com/u/dade.james)\
**Post date:** [March 16, 2022, 5:40pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/6 "2022-03-16T17:40:01Z")

</div>

I followed the instructions on the nvidia website.

I used the run file. NVIDIA-Linux-x86\_64-470.57.02.run

(base) [root@osmr7-gpuserver sa]# ls -l /usr/lib64/_nvidia_  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libEGL\_nvidia.so.0 → libEGL\_nvidia.so.470.57.02  
-rwxr-xr-x. 1 root root 1312784 Feb 17 10:01 /usr/lib64/libEGL\_nvidia.so.470.57.02  
lrwxrwxrwx. 1 root root 32 Feb 17 10:01 /usr/lib64/libGLESv1\_CM\_nvidia.so.1 → libGLESv1\_CM\_nvidia.so.470.57.02  
-rwxr-xr-x. 1 root root 67880 Feb 17 10:01 /usr/lib64/libGLESv1\_CM\_nvidia.so.470.57.02  
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libGLESv2\_nvidia.so.2 → libGLESv2\_nvidia.so.470.57.02  
-rwxr-xr-x. 1 root root 117032 Feb 17 10:01 /usr/lib64/libGLESv2\_nvidia.so.470.57.02  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libGLX\_nvidia.so.0 → libGLX\_nvidia.so.470.57.02  
-rwxr-xr-x. 1 root root 1289616 Feb 17 10:01 /usr/lib64/libGLX\_nvidia.so.470.57.02  
lrwxrwxrwx. 1 root root 24 Feb 17 10:01 /usr/lib64/libnvidia-allocator.so → libnvidia-allocator.so.1  
lrwxrwxrwx. 1 root root 32 Feb 17 10:01 /usr/lib64/libnvidia-allocator.so.1 → libnvidia-allocator.so.470.57.02  
-rwxr-xr-x. 1 root root 98688 Feb 17 10:01 /usr/lib64/libnvidia-allocator.so.470.57.02  
-rwxr-xr-x. 1 root root 762400 Feb 17 10:01 /usr/lib64/libnvidia-cbl.so.470.57.02  
lrwxrwxrwx. 1 root root 18 Feb 17 10:01 /usr/lib64/libnvidia-cfg.so → libnvidia-cfg.so.1  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-cfg.so.1 → libnvidia-cfg.so.470.57.02  
-rwxr-xr-x. 1 root root 221552 Feb 17 10:01 /usr/lib64/libnvidia-cfg.so.470.57.02  
-rwxr-xr-x. 1 root root 85507584 Jan 24 18:47 /usr/lib64/libnvidia-compiler-next.so.510.47.03  
-rwxr-xr-x. 1 root root 55972120 Feb 17 10:01 /usr/lib64/libnvidia-compiler.so.470.57.02  
lrwxrwxrwx. 1 root root 36 Feb 17 10:01 /usr/lib64/libnvidia-compiler.so.510.47.03 → libnvidia-compiler-next.so.510.47.03  
-rwxr-xr-x. 1 root root 30215216 Feb 17 10:01 /usr/lib64/libnvidia-eglcore.so.470.57.02  
lrwxrwxrwx. 1 root root 30 Feb 17 10:01 /usr/lib64/libnvidia-egl-wayland.so.1 → libnvidia-egl-wayland.so.1.1.7  
-rwxr-xr-x. 1 root root 43648 Feb 17 10:01 /usr/lib64/libnvidia-egl-wayland.so.1.1.7  
lrwxrwxrwx. 1 root root 21 Feb 17 10:01 /usr/lib64/libnvidia-encode.so → libnvidia-encode.so.1  
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libnvidia-encode.so.1 → libnvidia-encode.so.470.57.02  
-rwxr-xr-x. 1 root root 116768 Feb 17 10:01 /usr/lib64/libnvidia-encode.so.470.57.02  
lrwxrwxrwx. 1 root root 18 Feb 17 10:01 /usr/lib64/libnvidia-fbc.so → libnvidia-fbc.so.1  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-fbc.so.1 → libnvidia-fbc.so.470.57.02  
-rwxr-xr-x. 1 root root 129760 Feb 17 10:01 /usr/lib64/libnvidia-fbc.so.470.57.02  
-rwxr-xr-x. 1 root root 32038032 Feb 17 10:01 /usr/lib64/libnvidia-glcore.so.470.57.02  
-rwxr-xr-x. 1 root root 639848 Feb 17 10:01 /usr/lib64/libnvidia-glsi.so.470.57.02  
-rwxr-xr-x. 1 root root 13032888 Feb 17 10:01 /usr/lib64/libnvidia-glvkspirv.so.470.57.02  
-rwxr-xr-x. 1 root root 1367368 Feb 17 10:01 /usr/lib64/libnvidia-gtk2.so.470.57.02  
-rwxr-xr-x. 1 root root 1376040 Feb 17 10:01 /usr/lib64/libnvidia-gtk3.so.470.57.02  
lrwxrwxrwx. 1 root root 18 Feb 17 10:01 /usr/lib64/libnvidia-ifr.so → libnvidia-ifr.so.1  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-ifr.so.1 → libnvidia-ifr.so.470.57.02  
-rwxr-xr-x. 1 root root 219736 Feb 17 10:01 /usr/lib64/libnvidia-ifr.so.470.57.02  
lrwxrwxrwx. 1 root root 17 Feb 17 10:01 /usr/lib64/libnvidia-ml.so → libnvidia-ml.so.1  
lrwxrwxrwx. 1 root root 25 Feb 17 10:01 /usr/lib64/libnvidia-ml.so.1 → libnvidia-ml.so.470.57.02  
-rwxr-xr-x. 1 root root 1828056 Feb 17 10:01 /usr/lib64/libnvidia-ml.so.470.57.02  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-ngx.so.1 → libnvidia-ngx.so.470.57.02  
-rwxr-xr-x. 1 root root 3310184 Feb 17 10:01 /usr/lib64/libnvidia-ngx.so.470.57.02  
lrwxrwxrwx. 1 root root 19 Feb 17 10:01 /usr/lib64/libnvidia-nvvm.so → libnvidia-nvvm.so.4  
lrwxrwxrwx. 1 root root 23 Feb 17 10:01 /usr/lib64/libnvidia-nvvm.so.4 → libnvidia-nvvm.so.4.0.0  
-rwxr-xr-x. 1 root root 26659608 Feb 17 10:01 /usr/lib64/libnvidia-nvvm.so.4.0.0  
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libnvidia-opencl.so.1 → libnvidia-opencl.so.470.57.02  
-rwxr-xr-x. 1 root root 18224920 Feb 17 10:01 /usr/lib64/libnvidia-opencl.so.470.57.02  
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-opticalflow.so → libnvidia-opticalflow.so.1  
lrwxrwxrwx. 1 root root 34 Feb 17 10:01 /usr/lib64/libnvidia-opticalflow.so.1 → libnvidia-opticalflow.so.470.57.02  
-rwxr-xr-x. 1 root root 47096 Feb 17 10:01 /usr/lib64/libnvidia-opticalflow.so.470.57.02  
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libnvidia-ptxjitcompiler.so → libnvidia-ptxjitcompiler.so.1  
lrwxrwxrwx. 1 root root 37 Feb 17 10:01 /usr/lib64/libnvidia-ptxjitcompiler.so.1 → libnvidia-ptxjitcompiler.so.470.57.02  
-rwxr-xr-x. 1 root root 11144376 Feb 17 10:01 /usr/lib64/libnvidia-ptxjitcompiler.so.470.57.02  
-rwxr-xr-x. 1 root root 82662840 Feb 17 10:01 /usr/lib64/libnvidia-rtcore.so.470.57.02  
-rwxr-xr-x. 1 root root 18456 Feb 17 10:01 /usr/lib64/libnvidia-tls.so.470.57.02  
lrwxrwxrwx. 1 root root 34 Feb 17 10:01 /usr/lib64/libvdpau\_nvidia.so → vdpau/libvdpau\_nvidia.so.470.57.02

/usr/lib64/nvidia:  
total 0  
drwxr-xr-x. 2 root root 41 Feb 17 10:01 wine  
(base) [root@osmr7-gpuserver sa]#

I have run the nvidia-bug-report.sh script and would upload it to nvidia if I knew who to upload it to?

---

<div class="post-metadata">

**Author:** ![generix](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/generix/32/16024_2.png) [@generix](https://forums.developer.nvidia.com/u/generix)\
**Post date:** [March 17, 2022, 8:41am UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/7 "2022-03-17T08:41:29Z")

</div>

You installed the runfile driver over an already installed packaged driver, now you have a mix of both. Please uninstall both, then reinstall the repo driver. No need to use the runfile installer.

---

<div class="post-metadata">

**Author:** ![dade.james](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dade.james](https://forums.developer.nvidia.com/u/dade.james)\
**Post date:** [March 17, 2022, 12:13pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/8 "2022-03-17T12:13:19Z")

</div>

was looking for the nvidia-smi utility.

Need to find a way to uninstall the runfile installed packages. looking at the documentation.

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![generix](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/generix/32/16024_2.png) [@generix](https://forums.developer.nvidia.com/u/generix)\
**Post date:** [March 17, 2022, 12:18pm UTC](https://forums.developer.nvidia.com/t/failed-to-initialize-nvml-driver-library-version-mismatch-on-rhel8/208355/9 "2022-03-17T12:18:31Z")

</div>

Just run it again with --uninstall option.