CUDA Driver Failure after Redhat EL6 Kernel Update - What are the Nvidia/CUDA reinstallation steps?

Recently, we updated our CAE system kernel to:
Red Hat Enterprise Linux Server release 6.10 (Santiago)
Kernel 2.6.32-754.3.5.el6.x86_64 on an x86_64

Now our CUDA Drivers are broken. I am told by Redhat Technical Supoort that I need to re-install the Nvidia drivers and the cuda drivers.

What are the steps necessary to re-install the Nvidia drivers and re-install CUDA for the new kernel? Do I need specific drivers for this kernel?

Yum update cuda fails.

rpm -qa | grep nvid

xorg-x11-drv-nvidia-devel-340.29-1.el6.x86_64
xorg-x11-drv-nvidia-gl-340.29-1.el6.x86_64
xorg-x11-drv-nvidia-libs-340.29-1.el6.i686
xorg-x11-drv-nvidia-gl-340.29-1.el6.i686
xorg-x11-drv-nvidia-libs-340.29-1.el6.x86_64
xorg-x11-drv-nvidia-devel-340.29-1.el6.i686
xorg-x11-drv-nvidia-340.29-1.el6.x86_64
nvidia-uvm-kmod-340.29-3.el6.x86_64
nvidia-kmod-340.29-2.el6.x86_64

rpm -qa | grep cuda

cuda-cudart-dev-6-5-6.5-14.x86_64
cuda-core-6-5-6.5-14.x86_64
cuda-drivers-340.29-0.x86_64
cuda-cusparse-6-5-6.5-14.x86_64
cuda-visual-tools-6-5-6.5-14.x86_64
cuda-6.5-14.x86_64
cuda-cufft-dev-6-5-6.5-14.x86_64
cuda-curand-dev-6-5-6.5-14.x86_64
cuda-6-5-6.5-14.x86_64
cuda-misc-headers-6-5-6.5-14.x86_64
cuda-cublas-6-5-6.5-14.x86_64
cuda-documentation-6-5-6.5-14.x86_64
cuda-npp-dev-6-5-6.5-14.x86_64
cuda-command-line-tools-6-5-6.5-14.x86_64
cuda-samples-6-5-6.5-14.x86_64
cuda-cufft-6-5-6.5-14.x86_64
cuda-curand-6-5-6.5-14.x86_64
cuda-cusparse-dev-6-5-6.5-14.x86_64
cuda-driver-dev-6-5-6.5-14.x86_64
cuda-toolkit-6-5-6.5-14.x86_64
cuda-repo-rhel6-10.0.130-1.x86_64
cuda-npp-6-5-6.5-14.x86_64
cuda-runtime-6-5-6.5-14.x86_64
cuda-license-6-5-6.5-14.x86_64
cuda-cublas-dev-6-5-6.5-14.x86_64
cuda-cudart-6-5-6.5-14.x86_64

RHEL 6.1 is rather old. See if dkms is still installed, you should be able to reinstall the kernel modules running (as root)
/usr/lib/dkms/dkms_autoinstaller start
dracut -f
If that doesn’t work, please run nvidia-bug-report.sh as root and attach the resulting .gz file to your post. Hovering the mouse over an existing post of yours will reveal a paperclip icon.
https://devtalk.nvidia.com/default/topic/1043347/announcements/attaching-files-to-forum-topics-posts/

Hello, it seems that the dkms autoinstaller fails.

[root@norscae2 ~]# /usr/lib/dkms/dkms_autoinstaller start
dkms: running auto installation service for kernel 2.6.32-754.3.5.el6.x86_64
Error! echo
Your kernel headers for kernel 2.6.32-754.3.5.el6.x86_64 cannot be found at
/lib/modules/2.6.32-754.3.5.el6.x86_64/build or /lib/modules/2.6.32-754.3.5.el6.
x86_64/source.
Error! echo
Your kernel headers for kernel 2.6.32-754.3.5.el6.x86_64 cannot be found at
/lib/modules/2.6.32-754.3.5.el6.x86_64/build or /lib/modules/2.6.32-754.3.5.el6.
x86_64/source.
Error! echo
Your kernel headers for kernel 2.6.32-754.3.5.el6.x86_64 cannot be found at
/lib/modules/2.6.32-754.3.5.el6.x86_64/build or /lib/modules/2.6.32-754.3.5.el6.
x86_64/source.
Error! echo
Your kernel headers for kernel 2.6.32-754.3.5.el6.x86_64 cannot be found at
/lib/modules/2.6.32-754.3.5.el6.x86_64/build or /lib/modules/2.6.32-754.3.5.el6.
x86_64/source.
Done.
[root@norscae2 ~]#

rpm -qa shows that the dkms rpm is installed.

rpm -qa | grep dkms
dkms-2.3-1.20161202gitde1dca9.el6.noarch

The NVidia bug report is attached.

nvidia-bug-report.log.gz (2.85 MB)

Please install the packages kernel-devel and kernel-headers then try the procedure from my last post again.

both kernel-devel and kernel-headers were already installed. Nothing to do.

[root@norscae2 ~]# yum install kernel-devel
Loaded plugins: aliases, changelog, presto, product-id, rhnplugin, search-
: disabled-repos, security, subscription-manager, tmprepo, verify,
: versionlock
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Package kernel-devel-2.6.32-754.6.3.el6.x86_64 already installed and latest vers
ion
Nothing to do

[root@norscae2 ~]# yum install kernel-headers
Loaded plugins: aliases, changelog, presto, product-id, rhnplugin, search-
: disabled-repos, security, subscription-manager, tmprepo, verify,
: versionlock
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Package kernel-headers-2.6.32-754.6.3.el6.x86_64 already installed and latest ve
rsion
Nothing to do

Please run
dracut -f
and reboot. Create a new nvidia-bug-report.log afterwards.

Done. Ran dracut -f and rebooted. nvidia-bug-report.log2.gz is attached.

nvidia-bug-report.log2.gz (2.79 MB)

You’re running an old kernel (2.6.32-754.3.5) but the installed headers are for the current kernel (2.6.32-754.6.3), so please either change to the new kernel (6.3) probably at the boot prompt or install the headers for the currently running kernel (3.5)

Thank you. It was necessary to fallback the kernel to an older version due to a problem with our backup software. We’ll update the kernel, as soon as the backup software will support the new kernel version, and then we’ll make sure the kernel headers matches the kernel version. Please close out this thread.