Failed to initialize NVML: Driver/library version mismatch on RHEL8

Getting this error when trying to run nvidia-smi on a RHEL 8.5 server

NVRM version: NVIDIA UNIX x86_64 Kernel Module 510.47.03 is my current driver.

01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eH3 (rev 02) (prog-if 00 [VGA controller])
Subsystem: Hewlett Packard Enterprise iLO5 VGA
8e:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])
9c:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])
ce:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])
dc:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1) (prog-if 00 [VGA controller])

[root@osmr7-gpuserver sa]# rpm -qa | grep -i nvidia
nvidia-persistenced-510.47.03-1.el8.x86_64
nvidia-driver-devel-510.47.03-1.el8.x86_64
nvidia-xconfig-510.47.03-1.el8.x86_64
nvidia-driver-cuda-libs-510.47.03-1.el8.x86_64
nvidia-driver-510.47.03-1.el8.x86_64
nvidia-modprobe-510.47.03-1.el8.x86_64
nvidia-driver-NVML-510.47.03-1.el8.x86_64
nvidia-libXNVCtrl-devel-510.47.03-1.el8.x86_64
dnf-plugin-nvidia-2.0-1.el8.noarch
nvidia-libXNVCtrl-510.47.03-1.el8.x86_64
nvidia-driver-NvFBCOpenGL-510.47.03-1.el8.x86_64
nvidia-kmod-common-510.47.03-1.el8.noarch
nvidia-settings-510.47.03-1.el8.x86_64
nvidia-driver-libs-510.47.03-1.el8.x86_64
kmod-nvidia-latest-dkms-510.47.03-1.el8.x86_64
nvidia-driver-cuda-510.47.03-1.el8.x86_64

Please check for leftovers from an earlier driver install,
ls -l /usr/lib64/libcuda*

(base) [root@osmr7-gpuserver sa]# ls -l /usr/lib64/libcuda*
lrwxrwxrwx. 1 root root 12 Feb 17 10:01 /usr/lib64/libcuda.so → libcuda.so.1
lrwxrwxrwx. 1 root root 20 Feb 17 10:01 /usr/lib64/libcuda.so.1 → libcuda.so.470.57.02
-rwxr-xr-x. 1 root root 22267208 Feb 17 10:01 /usr/lib64/libcuda.so.470.57.02

(base) [root@osmr7-gpuserver sa]# rpm -qa | grep -i cuda
cuda-nvvp-11-6-11.6.112-1.x86_64
cuda-memcheck-11-6-11.6.112-1.x86_64
cuda-cuxxfilt-11-6-11.6.112-1.x86_64
cuda-compiler-11-6-11.6.1-1.x86_64
cuda-nvml-devel-11-6-11.6.55-1.x86_64
nvidia-driver-cuda-libs-510.47.03-1.el8.x86_64
cuda-libraries-devel-11-6-11.6.1-1.x86_64
cuda-visual-tools-11-6-11.6.1-1.x86_64
cuda-toolkit-11-6-config-common-11.6.55-1.noarch
cuda-cudart-devel-11-6-11.6.55-1.x86_64
cuda-driver-devel-11-6-11.6.55-1.x86_64
cuda-nvrtc-11-6-11.6.112-1.x86_64
cuda-nvprune-11-6-11.6.112-1.x86_64
cuda-cupti-11-6-11.6.112-1.x86_64
cuda-tools-11-6-11.6.1-1.x86_64
cuda-repo-rhel8-11-6-local-11.6.0_510.39.01-1.x86_64
cuda-nvdisasm-11-6-11.6.104-1.x86_64
cuda-nsight-11-6-11.6.112-1.x86_64
cuda-11.6.1-1.x86_64
cuda-toolkit-config-common-11.6.55-1.noarch
cuda-nvrtc-devel-11-6-11.6.112-1.x86_64
cuda-documentation-11-6-11.6.112-1.x86_64
cuda-sanitizer-11-6-11.6.112-1.x86_64
cuda-cudart-11-6-11.6.55-1.x86_64
cuda-samples-11-6-11.6.101-1.x86_64
cuda-libraries-11-6-11.6.1-1.x86_64
cuda-nvcc-11-6-11.6.112-1.x86_64
cuda-cuobjdump-11-6-11.6.112-1.x86_64
cuda-toolkit-11-6-11.6.1-1.x86_64
cuda-demo-suite-11-6-11.6.55-1.x86_64
cuda-drivers-510.47.03-1.x86_64
cuda-nsight-compute-11-6-11.6.1-1.x86_64
cuda-toolkit-11-config-common-11.6.55-1.noarch
cuda-cccl-11-6-11.6.55-1.x86_64
cuda-nvtx-11-6-11.6.112-1.x86_64
cuda-command-line-tools-11-6-11.6.1-1.x86_64
cuda-nvprof-11-6-11.6.112-1.x86_64
cuda-nsight-systems-11-6-11.6.1-1.x86_64
cuda-11-6-11.6.1-1.x86_64
nvidia-driver-cuda-510.47.03-1.el8.x86_64
cuda-runtime-11-6-11.6.1-1.x86_64
cuda-gdb-11-6-11.6.112-1.x86_64
(base) [root@osmr7-gpuserver sa]#

Looks like you have leftovers from an ealier 470 driver install. Did you previously use the runfile installer? Please also check for more leftovers
ls -l /usr/lib64/*nvidia*

I followed the instructions on the nvidia website.

I used the run file. NVIDIA-Linux-x86_64-470.57.02.run

(base) [root@osmr7-gpuserver sa]# ls -l /usr/lib64/nvidia
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libEGL_nvidia.so.0 → libEGL_nvidia.so.470.57.02
-rwxr-xr-x. 1 root root 1312784 Feb 17 10:01 /usr/lib64/libEGL_nvidia.so.470.57.02
lrwxrwxrwx. 1 root root 32 Feb 17 10:01 /usr/lib64/libGLESv1_CM_nvidia.so.1 → libGLESv1_CM_nvidia.so.470.57.02
-rwxr-xr-x. 1 root root 67880 Feb 17 10:01 /usr/lib64/libGLESv1_CM_nvidia.so.470.57.02
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libGLESv2_nvidia.so.2 → libGLESv2_nvidia.so.470.57.02
-rwxr-xr-x. 1 root root 117032 Feb 17 10:01 /usr/lib64/libGLESv2_nvidia.so.470.57.02
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libGLX_nvidia.so.0 → libGLX_nvidia.so.470.57.02
-rwxr-xr-x. 1 root root 1289616 Feb 17 10:01 /usr/lib64/libGLX_nvidia.so.470.57.02
lrwxrwxrwx. 1 root root 24 Feb 17 10:01 /usr/lib64/libnvidia-allocator.so → libnvidia-allocator.so.1
lrwxrwxrwx. 1 root root 32 Feb 17 10:01 /usr/lib64/libnvidia-allocator.so.1 → libnvidia-allocator.so.470.57.02
-rwxr-xr-x. 1 root root 98688 Feb 17 10:01 /usr/lib64/libnvidia-allocator.so.470.57.02
-rwxr-xr-x. 1 root root 762400 Feb 17 10:01 /usr/lib64/libnvidia-cbl.so.470.57.02
lrwxrwxrwx. 1 root root 18 Feb 17 10:01 /usr/lib64/libnvidia-cfg.so → libnvidia-cfg.so.1
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-cfg.so.1 → libnvidia-cfg.so.470.57.02
-rwxr-xr-x. 1 root root 221552 Feb 17 10:01 /usr/lib64/libnvidia-cfg.so.470.57.02
-rwxr-xr-x. 1 root root 85507584 Jan 24 18:47 /usr/lib64/libnvidia-compiler-next.so.510.47.03
-rwxr-xr-x. 1 root root 55972120 Feb 17 10:01 /usr/lib64/libnvidia-compiler.so.470.57.02
lrwxrwxrwx. 1 root root 36 Feb 17 10:01 /usr/lib64/libnvidia-compiler.so.510.47.03 → libnvidia-compiler-next.so.510.47.03
-rwxr-xr-x. 1 root root 30215216 Feb 17 10:01 /usr/lib64/libnvidia-eglcore.so.470.57.02
lrwxrwxrwx. 1 root root 30 Feb 17 10:01 /usr/lib64/libnvidia-egl-wayland.so.1 → libnvidia-egl-wayland.so.1.1.7
-rwxr-xr-x. 1 root root 43648 Feb 17 10:01 /usr/lib64/libnvidia-egl-wayland.so.1.1.7
lrwxrwxrwx. 1 root root 21 Feb 17 10:01 /usr/lib64/libnvidia-encode.so → libnvidia-encode.so.1
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libnvidia-encode.so.1 → libnvidia-encode.so.470.57.02
-rwxr-xr-x. 1 root root 116768 Feb 17 10:01 /usr/lib64/libnvidia-encode.so.470.57.02
lrwxrwxrwx. 1 root root 18 Feb 17 10:01 /usr/lib64/libnvidia-fbc.so → libnvidia-fbc.so.1
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-fbc.so.1 → libnvidia-fbc.so.470.57.02
-rwxr-xr-x. 1 root root 129760 Feb 17 10:01 /usr/lib64/libnvidia-fbc.so.470.57.02
-rwxr-xr-x. 1 root root 32038032 Feb 17 10:01 /usr/lib64/libnvidia-glcore.so.470.57.02
-rwxr-xr-x. 1 root root 639848 Feb 17 10:01 /usr/lib64/libnvidia-glsi.so.470.57.02
-rwxr-xr-x. 1 root root 13032888 Feb 17 10:01 /usr/lib64/libnvidia-glvkspirv.so.470.57.02
-rwxr-xr-x. 1 root root 1367368 Feb 17 10:01 /usr/lib64/libnvidia-gtk2.so.470.57.02
-rwxr-xr-x. 1 root root 1376040 Feb 17 10:01 /usr/lib64/libnvidia-gtk3.so.470.57.02
lrwxrwxrwx. 1 root root 18 Feb 17 10:01 /usr/lib64/libnvidia-ifr.so → libnvidia-ifr.so.1
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-ifr.so.1 → libnvidia-ifr.so.470.57.02
-rwxr-xr-x. 1 root root 219736 Feb 17 10:01 /usr/lib64/libnvidia-ifr.so.470.57.02
lrwxrwxrwx. 1 root root 17 Feb 17 10:01 /usr/lib64/libnvidia-ml.so → libnvidia-ml.so.1
lrwxrwxrwx. 1 root root 25 Feb 17 10:01 /usr/lib64/libnvidia-ml.so.1 → libnvidia-ml.so.470.57.02
-rwxr-xr-x. 1 root root 1828056 Feb 17 10:01 /usr/lib64/libnvidia-ml.so.470.57.02
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-ngx.so.1 → libnvidia-ngx.so.470.57.02
-rwxr-xr-x. 1 root root 3310184 Feb 17 10:01 /usr/lib64/libnvidia-ngx.so.470.57.02
lrwxrwxrwx. 1 root root 19 Feb 17 10:01 /usr/lib64/libnvidia-nvvm.so → libnvidia-nvvm.so.4
lrwxrwxrwx. 1 root root 23 Feb 17 10:01 /usr/lib64/libnvidia-nvvm.so.4 → libnvidia-nvvm.so.4.0.0
-rwxr-xr-x. 1 root root 26659608 Feb 17 10:01 /usr/lib64/libnvidia-nvvm.so.4.0.0
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libnvidia-opencl.so.1 → libnvidia-opencl.so.470.57.02
-rwxr-xr-x. 1 root root 18224920 Feb 17 10:01 /usr/lib64/libnvidia-opencl.so.470.57.02
lrwxrwxrwx. 1 root root 26 Feb 17 10:01 /usr/lib64/libnvidia-opticalflow.so → libnvidia-opticalflow.so.1
lrwxrwxrwx. 1 root root 34 Feb 17 10:01 /usr/lib64/libnvidia-opticalflow.so.1 → libnvidia-opticalflow.so.470.57.02
-rwxr-xr-x. 1 root root 47096 Feb 17 10:01 /usr/lib64/libnvidia-opticalflow.so.470.57.02
lrwxrwxrwx. 1 root root 29 Feb 17 10:01 /usr/lib64/libnvidia-ptxjitcompiler.so → libnvidia-ptxjitcompiler.so.1
lrwxrwxrwx. 1 root root 37 Feb 17 10:01 /usr/lib64/libnvidia-ptxjitcompiler.so.1 → libnvidia-ptxjitcompiler.so.470.57.02
-rwxr-xr-x. 1 root root 11144376 Feb 17 10:01 /usr/lib64/libnvidia-ptxjitcompiler.so.470.57.02
-rwxr-xr-x. 1 root root 82662840 Feb 17 10:01 /usr/lib64/libnvidia-rtcore.so.470.57.02
-rwxr-xr-x. 1 root root 18456 Feb 17 10:01 /usr/lib64/libnvidia-tls.so.470.57.02
lrwxrwxrwx. 1 root root 34 Feb 17 10:01 /usr/lib64/libvdpau_nvidia.so → vdpau/libvdpau_nvidia.so.470.57.02

/usr/lib64/nvidia:
total 0
drwxr-xr-x. 2 root root 41 Feb 17 10:01 wine
(base) [root@osmr7-gpuserver sa]#

I have run the nvidia-bug-report.sh script and would upload it to nvidia if I knew who to upload it to?

You installed the runfile driver over an already installed packaged driver, now you have a mix of both. Please uninstall both, then reinstall the repo driver. No need to use the runfile installer.

was looking for the nvidia-smi utility.

Need to find a way to uninstall the runfile installed packages. looking at the documentation.

Thanks for your help.

Just run it again with --uninstall option.