Nvidia driver not booting

NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev a1)
Rocky Linux 9.1
Alienware R8 with latest 1.0.26 BIOS.

Detailed description:
5.14.0-162.12.1.el9_1.0.1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 23 19:25:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Boots but doesn’t reach GUI after:
dnf module install nvidia-driver:515
dnf module install nvidia-driver:520
dnf module install nvidia-driver:525

Logs attached.
nvidia-bug-report.log.gz (85.6 KB)
nvidia-bug-report.log.gz (85.7 KB)
nvidia-bug-report.log.gz (85.8 KB)

We’ll try to troubleshoot some more with other kernels, etc…

Hi @mabarkdoll

Is there a Rocky kernel version that does work on your machine? Example in Rocky’s vault:

sudo dnf config-manager --add-repo https://download.rockylinux.org/vault/rocky/9.0/BaseOS/x86_64/
sudo dnf info --showduplicates kernel-core | grep -e Version -e Release | awk '{print $NF}' | paste -sd '-\n'
5.14.0-162.12.1.el9_1.0.2
5.14.0-70.30.1.el9_0

Looks like 5.14.0-70.30.1.el9_0 is common between RHEL 9 and RockyLinux 9.

Can you share the output from dnf module install ?

I have secureboot enabled and the machine owner-key installed with UEFI, so I haven’t had to use DKMS in the past.

# mokutil --list-enrolled
[key 1]
SHA1 Fingerprint: 0e:2a:bf:72:66:32:95:3a:d2:05:b3:cd:c7:eb:24:15:8b:31:b3:bb
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            10:6c:57:1c:ee:af:29:24:c0:aa:85:e5:aa:b4:85:42:9a:b1:d8:86
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=Delaware, L=Dover, O=Rocky Enterprise Software Foundation, OU=Release engineering team, CN=Rocky Linux Secure Boot Root CA/emailAddress=security@rockylinux.org
        Validity
            Not Before: Jun 20 15:05:01 2021 GMT
            Not After : Jun 18 15:05:01 2031 GMT
        Subject: C=US, ST=Delaware, L=Dover, O=Rocky Enterprise Software Foundation, OU=Release engineering team, CN=Rocky Linux Secure Boot Root CA/emailAddress=security@rockylinux.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e3:c4:6b:0f:5b:08:fd:db:51:fc:e0:49:39:e9:
                    a8:53:9b:1e:f0:57:ae:da:2c:05:33:5f:6f:2d:a4:
                    76:9e:db:63:40:86:77:a6:20:4b:35:c5:75:ca:dc:
                    5c:4b:a5:f8:ea:1a:e1:c2:47:fa:d3:c2:ef:fe:52:
                    83:94:16:42:90:b0:f2:13:2b:aa:73:22:e8:a0:42:
                    1b:35:02:a8:03:df:10:49:bd:a0:3d:94:35:4e:44:
                    25:78:08:44:99:c0:23:58:63:cc:56:48:a0:40:8b:
                    cb:eb:a0:fd:fc:93:22:46:df:43:36:69:01:f0:1d:
                    b1:7f:62:31:a0:50:be:56:f7:78:3e:2e:22:84:6f:
                    de:8a:81:c1:49:b1:94:7e:fb:7c:d8:f7:4a:20:1e:
                    37:9e:91:05:e8:5f:bf:aa:4e:cd:67:c9:cc:9e:5d:
                    17:90:7f:fd:74:64:c4:a9:e3:1b:03:03:d2:35:a2:
                    87:2a:e4:e5:74:b8:c3:56:82:71:0c:b3:e3:cc:38:
                    a3:51:bb:58:b9:97:be:36:95:9e:64:32:25:d5:a4:
                    94:9d:01:34:5e:cf:68:39:9e:56:74:72:d4:22:1b:
                    34:46:44:14:c2:14:17:88:c3:16:ac:ea:9a:f9:85:
                    f1:00:fb:d9:0e:a0:e0:cf:d1:3d:2c:24:d2:59:94:
                    a4:a5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                4C:2C:6B:D7:D6:4E:E8:15:81:CA:B8:E9:86:66:1F:65:E2:16:6F:C4
            X509v3 Authority Key Identifier: 
                4C:2C:6B:D7:D6:4E:E8:15:81:CA:B8:E9:86:66:1F:65:E2:16:6F:C4
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Key Usage: 
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        4f:b2:73:90:ad:13:07:fb:1f:66:05:65:8d:21:7e:a9:48:d3:
        92:fc:2e:14:04:9b:ff:48:7d:74:d6:89:12:af:eb:bb:51:91:
        43:9a:df:2b:fe:8c:53:9c:4a:86:96:0a:7b:d4:78:b2:b7:bf:
        c6:61:5f:d9:0c:b6:40:d4:f1:b2:1e:14:74:10:da:42:3f:a5:
        95:4b:22:e4:ec:fb:a9:6a:c9:14:c5:eb:02:f9:04:08:f8:6f:
        51:ed:48:9d:d6:c0:14:1e:00:42:d5:cb:2c:a5:a8:9e:30:8c:
        73:8c:c9:b0:d0:75:82:52:b8:ee:f2:4b:87:18:a0:09:ae:93:
        11:61:ed:62:8d:8c:cb:9e:59:05:b2:41:68:67:d6:5b:fd:17:
        6e:f4:87:59:bb:bd:d7:14:07:ca:ea:7b:45:1a:f1:fe:f5:6b:
        08:78:50:ab:29:a2:d4:41:35:96:ba:76:89:d7:53:5e:46:01:
        55:4f:0c:44:2a:7e:7d:78:49:85:2b:08:29:b5:cc:e5:f4:f4:
        78:1f:64:32:e0:9d:e3:a2:9f:0d:a6:d3:7f:29:f8:89:bb:e2:
        7d:19:65:54:3f:42:e5:a9:4b:2e:31:98:9a:2a:19:0e:b7:ed:
        c3:f9:4a:ca:63:b8:43:c3:93:7b:bb:24:50:1a:15:48:54:c3:
        9a:2a:07:33

[key 2]
SHA1 Fingerprint: d5:6f:7e:6a:ce:7e:71:7d:06:07:aa:40:73:11:c8:9b:bc:3b:b5:e9
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            55:7c:fd:cc:d0:ea:36:2f:0c:09:ba:00:a1:6c:b1:b8:e6:c1:eb:ab
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=NVIDIA, CN=NVIDIA/emailAddress=sw-cuda-installer@nvidia.com
        Validity
            Not Before: Feb  8 01:23:48 2019 GMT
            Not After : Jan 15 01:23:48 2119 GMT
        Subject: O=NVIDIA, CN=NVIDIA/emailAddress=sw-cuda-installer@nvidia.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:dc:60:57:69:90:81:c2:dc:b2:8b:05:84:19:1d:
                    6f:18:4a:d1:6f:dc:0b:00:23:85:ff:31:eb:2f:04:
                    e8:77:f3:78:67:c5:28:81:df:57:00:11:63:80:14:
                    b8:b9:cd:09:27:28:9d:e2:da:cc:6c:eb:f8:41:41:
                    e3:de:77:b5:90:7f:c7:79:f8:a7:f3:0f:19:07:b2:
                    24:2e:c9:0f:db:f9:f4:e9:2c:a1:45:3b:2d:ec:86:
                    37:d2:e9:3a:b5:fc:cf:cc:49:2a:77:f1:b3:ff:95:
                    21:3c:67:03:ae:89:b9:2e:7e:d1:4a:38:63:e6:43:
                    e5:0f:5e:4d:db:c4:96:30:01:f6:80:f3:1a:b6:82:
                    d4:a9:9d:cd:c5:0c:2e:55:98:b6:c0:ec:fe:c1:84:
                    53:d4:5c:fc:22:5f:58:d6:a1:7d:c7:0a:81:cf:0d:
                    17:57:e2:a1:4d:ec:e5:30:7f:64:10:e5:aa:f8:af:
                    6e:ad:19:95:f8:ba:d1:7c:7d:6d:1c:fe:c2:ad:ff:
                    c8:40:4b:bb:e6:10:1e:c0:49:69:75:0e:d2:e2:f7:
                    ac:30:b1:96:c4:ce:f5:7b:dd:9a:5e:70:6a:98:c3:
                    e7:b9:3c:65:43:67:12:e7:93:f6:76:79:6d:bb:19:
                    75:63:e4:43:9e:c7:b4:19:82:c0:39:af:a1:b8:b6:
                    75:c8:0a:bf:e7:33:33:fa:18:5f:cc:18:cd:40:06:
                    d3:66:fd:1f:f1:9c:16:b6:53:a4:96:e8:06:7a:95:
                    32:61:62:63:06:e7:98:fa:07:9d:3e:74:f2:4f:8e:
                    43:19:31:1e:dd:e1:22:a1:97:88:de:f1:19:06:2c:
                    94:53:87:84:53:c3:c3:56:9e:95:3a:5f:3a:8e:d9:
                    d0:c1:3b:ee:be:3b:7f:6b:06:ea:cb:f9:fc:fc:43:
                    54:a7:36:e5:8f:91:4d:ca:c6:37:ef:92:4b:cc:ba:
                    3a:49:42:9c:c0:3a:9a:18:42:17:95:64:78:c1:b6:
                    c3:e3:02:9e:10:33:a3:51:73:86:93:77:52:e0:f7:
                    d3:3a:a6:ce:01:07:a5:7b:93:81:a8:77:e6:4c:9b:
                    f9:03:7b:94:e5:6a:d6:4e:a0:9f:6c:d2:f4:01:cd:
                    6c:8b:fc:3e:56:ff:b0:05:a0:9b:18:1b:79:8f:03:
                    e0:1e:27:ab:dd:68:c0:26:70:38:fa:5d:21:f2:26:
                    c7:ac:10:e5:ca:5f:df:6a:9b:ca:6a:4d:72:a1:87:
                    68:88:b3:6c:8f:03:d1:ae:0c:b1:ad:d5:29:b3:2f:
                    40:d9:c6:d8:1d:97:d1:74:5b:8e:4b:f2:28:12:bc:
                    00:2f:fe:00:40:6c:d7:b3:eb:09:90:74:09:a7:c1:
                    02:3a:e7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature
            X509v3 Subject Key Identifier: 
                EC:03:1F:93:38:08:17:3B:AB:38:A0:F4:89:11:B5:1C:F1:6C:81:A0
            X509v3 Authority Key Identifier: 
                EC:03:1F:93:38:08:17:3B:AB:38:A0:F4:89:11:B5:1C:F1:6C:81:A0
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        22:c0:b5:90:45:88:87:ef:a5:51:fa:38:38:e0:70:77:91:ae:
        85:8a:a9:6d:41:2e:0e:58:c4:b2:b1:ec:96:25:b0:33:80:b2:
        6f:ec:25:c9:90:86:de:92:14:3f:35:98:a9:62:60:de:74:00:
        e0:ab:9e:44:cd:32:78:d1:00:27:d1:90:32:69:36:5d:81:6f:
        83:fb:80:5b:54:ae:42:3d:29:bf:ea:f5:d9:65:1a:c9:d5:e3:
        1d:89:9f:5c:0d:5e:e8:05:ac:b1:59:09:6a:7e:b8:85:05:59:
        2e:a6:19:08:4d:09:44:45:40:95:7e:d5:35:d2:5e:72:44:16:
        eb:80:5c:53:b6:0e:70:ac:0f:9d:dc:88:ae:48:47:91:11:46:
        21:95:21:57:ac:4f:ea:d5:4a:f6:e6:ee:13:02:0a:73:80:fe:
        1b:38:6c:c5:13:24:6b:3c:e8:5c:19:34:47:68:cc:05:ea:52:
        65:1c:3f:57:8f:c2:a6:d7:68:c9:5a:76:a0:f0:c8:be:54:59:
        da:23:14:61:fb:1a:89:47:3a:1e:3c:63:c7:64:b0:82:c8:eb:
        00:77:e4:20:09:4f:a8:f7:b1:70:c2:19:52:5e:fb:e6:35:06:
        19:69:db:68:0e:ff:ec:f8:2b:12:d3:2b:ee:b9:05:26:ae:a3:
        0c:5a:c4:46:95:7b:a7:88:a3:a0:cb:e9:d7:f0:1b:2a:2b:74:
        0f:c5:6f:0b:d0:ec:f6:65:61:f4:19:50:a9:ae:c8:eb:db:6d:
        a5:78:7e:80:4f:43:10:64:21:ed:aa:09:92:59:13:a6:c6:ab:
        b8:9a:95:f9:c0:94:b2:e1:88:4b:43:8c:46:da:81:89:a1:d7:
        2e:b7:f7:fb:77:7f:fc:cb:15:28:46:73:83:5d:bf:3a:52:35:
        26:b1:2d:78:3b:03:01:3d:3b:cd:d4:1b:46:8a:5e:52:3d:57:
        f5:04:48:33:7a:14:1f:e8:5f:27:59:ca:25:4d:12:0f:7b:20:
        4c:bc:c2:d9:1b:8e:11:60:97:14:c5:6f:05:17:86:ac:d0:3b:
        39:e2:da:f3:84:16:47:da:30:3b:74:a7:e9:0c:be:bf:93:49:
        93:f5:2e:7b:56:20:60:4e:34:4d:71:a9:a0:e0:6a:dc:54:3d:
        53:3e:50:52:6b:96:a0:44:2d:f1:1a:e6:87:d6:5e:8d:cd:ad:
        91:50:00:9e:82:b7:26:87:90:1d:0a:79:82:17:d8:f6:9c:12:
        e0:cd:02:99:32:f4:88:0f:82:67:04:3d:15:97:ea:f5:59:6f:
        e6:5e:6e:21:b3:e3:6b:0b:83:e0:88:84:37:31:63:80:41:b1:
        49:01:ae:6c:b2:96:51:27

Using latest kernel:

# uname -a 
Linux hostname.fqdn.edu 5.14.0-162.12.1.el9_1.0.2.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 30 22:14:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
# dnf module install nvidia-driver:latest
Last metadata expiration check: 0:08:21 ago on Mon 06 Feb 2023 09:49:02 AM CST.
Dependencies resolved.
================================================================================================================================================================
 Package                                                Architecture            Version                                Repository                          Size
================================================================================================================================================================
Installing group/module packages:
 cuda-drivers                                           x86_64                  525.85.12-1                            cuda-rhel9-x86_64                  8.1 k
 nvidia-driver                                          x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   43 M
 nvidia-driver-NVML                                     x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                  495 k
 nvidia-driver-NvFBCOpenGL                              x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   52 k
 nvidia-driver-cuda                                     x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                  484 k
 nvidia-driver-cuda-libs                                x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   54 M
 nvidia-driver-devel                                    x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                  9.6 k
 nvidia-driver-libs                                     x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                  187 M
 nvidia-kmod-common                                     noarch                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   12 k
 nvidia-libXNVCtrl                                      x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   52 k
 nvidia-libXNVCtrl-devel                                x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   51 k
 nvidia-modprobe                                        x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   28 k
 nvidia-persistenced                                    x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   33 k
 nvidia-settings                                        x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                  1.7 M
 nvidia-xconfig                                         x86_64                  3:525.85.12-1.el9                      cuda-rhel9-x86_64                   96 k
Installing dependencies:
 dnf-plugin-nvidia                                      noarch                  2.0-1.el9                              cuda-rhel9-x86_64                   13 k
 egl-wayland                                            x86_64                  1.1.9-2.el9                            appstream                           36 k
 kmod-nvidia-525.85.12-5.14.0-162.12.1                  x86_64                  3:525.85.12-3.el9_1                    cuda-rhel9-x86_64                   43 M
 libX11-devel                                           x86_64                  1.7.0-7.el9                            appstream                          941 k
 libXau-devel                                           x86_64                  1.0.9-8.el9                            appstream                           13 k
 libvdpau                                               x86_64                  1.4-6.el9                              appstream                           16 k
 libxcb-devel                                           x86_64                  1.13.1-9.el9                           appstream                          1.0 M
 ocl-icd                                                x86_64                  2.2.13-4.el9                           appstream                           51 k
 opencl-filesystem                                      noarch                  1.0-15.el9                             appstream                          7.4 k
 xorg-x11-proto-devel                                   noarch                  2021.4-2.el9                           appstream                          262 k
Installing module profiles:
 nvidia-driver/default                                                                                                                                         
Enabling module streams:
 nvidia-driver                                                                  latest                                                                         

Transaction Summary
================================================================================================================================================================
Install  25 Packages

Total download size: 332 M
Installed size: 813 M
Is this ok [y/N]: y
Downloading Packages:
(1/25): dnf-plugin-nvidia-2.0-1.el9.noarch.rpm                                                                                   65 kB/s |  13 kB     00:00    
(2/25): cuda-drivers-525.85.12-1.x86_64.rpm                                                                                      38 kB/s | 8.1 kB     00:00    
(3/25): nvidia-driver-NVML-525.85.12-1.el9.x86_64.rpm                                                                           4.7 MB/s | 495 kB     00:00    
(4/25): nvidia-driver-NvFBCOpenGL-525.85.12-1.el9.x86_64.rpm                                                                    381 kB/s |  52 kB     00:00    
(5/25): nvidia-driver-cuda-525.85.12-1.el9.x86_64.rpm                                                                           2.1 MB/s | 484 kB     00:00    
(6/25): kmod-nvidia-525.85.12-5.14.0-162.12.1-525.85.12-3.el9_1.x86_64.rpm                                                       41 MB/s |  43 MB     00:01    
(7/25): nvidia-driver-devel-525.85.12-1.el9.x86_64.rpm                                                                           52 kB/s | 9.6 kB     00:00    
(8/25): nvidia-driver-525.85.12-1.el9.x86_64.rpm                                                                                9.8 MB/s |  43 MB     00:04    
(9/25): nvidia-kmod-common-525.85.12-1.el9.noarch.rpm                                                                           103 kB/s |  12 kB     00:00    
(10/25): nvidia-libXNVCtrl-525.85.12-1.el9.x86_64.rpm                                                                           490 kB/s |  52 kB     00:00    
(11/25): nvidia-libXNVCtrl-devel-525.85.12-1.el9.x86_64.rpm                                                                     504 kB/s |  51 kB     00:00    
(12/25): nvidia-modprobe-525.85.12-1.el9.x86_64.rpm                                                                             275 kB/s |  28 kB     00:00    
(13/25): nvidia-persistenced-525.85.12-1.el9.x86_64.rpm                                                                         331 kB/s |  33 kB     00:00    
(14/25): nvidia-settings-525.85.12-1.el9.x86_64.rpm                                                                             6.9 MB/s | 1.7 MB     00:00    
(15/25): nvidia-xconfig-525.85.12-1.el9.x86_64.rpm                                                                              849 kB/s |  96 kB     00:00    
(16/25): xorg-x11-proto-devel-2021.4-2.el9.noarch.rpm                                                                           765 kB/s | 262 kB     00:00    
(17/25): nvidia-driver-cuda-libs-525.85.12-1.el9.x86_64.rpm                                                                      10 MB/s |  54 MB     00:05    
(18/25): egl-wayland-1.1.9-2.el9.x86_64.rpm                                                                                     180 kB/s |  36 kB     00:00    
(19/25): libXau-devel-1.0.9-8.el9.x86_64.rpm                                                                                    293 kB/s |  13 kB     00:00    
(20/25): opencl-filesystem-1.0-15.el9.noarch.rpm                                                                                 71 kB/s | 7.4 kB     00:00    
(21/25): libvdpau-1.4-6.el9.x86_64.rpm                                                                                          288 kB/s |  16 kB     00:00    
(22/25): libxcb-devel-1.13.1-9.el9.x86_64.rpm                                                                                   6.7 MB/s | 1.0 MB     00:00    
(23/25): ocl-icd-2.2.13-4.el9.x86_64.rpm                                                                                        492 kB/s |  51 kB     00:00    
(24/25): libX11-devel-1.7.0-7.el9.x86_64.rpm                                                                                    3.4 MB/s | 941 kB     00:00    
(25/25): nvidia-driver-libs-525.85.12-1.el9.x86_64.rpm                                                                           20 MB/s | 187 MB     00:09    
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                            31 MB/s | 332 MB     00:10     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: cuda-drivers-525.85.12-1.x86_64                                                                                                        1/1 
  Preparing        :                                                                                                                                        1/1 
  Installing       : nvidia-libXNVCtrl-3:525.85.12-1.el9.x86_64                                                                                            1/25 
  Installing       : nvidia-driver-NVML-3:525.85.12-1.el9.x86_64                                                                                           2/25 
  Running scriptlet: nvidia-driver-NVML-3:525.85.12-1.el9.x86_64                                                                                           2/25 
  Installing       : kmod-nvidia-525.85.12-5.14.0-162.12.1-3:525.85.12-3.el9_1.x86_64                                                                      3/25 
  Running scriptlet: kmod-nvidia-525.85.12-5.14.0-162.12.1-3:525.85.12-3.el9_1.x86_64                                                                      3/25 
depmod: WARNING: could not open modules.order at /lib/modules/5.14.0-162.12.1.el9_1.x86_64: No such file or directory
depmod: WARNING: could not open modules.builtin at /lib/modules/5.14.0-162.12.1.el9_1.x86_64: No such file or directory

  Installing       : libvdpau-1.4-6.el9.x86_64                                                                                                             4/25 
  Installing       : egl-wayland-1.1.9-2.el9.x86_64                                                                                                        5/25 
  Installing       : nvidia-driver-libs-3:525.85.12-1.el9.x86_64                                                                                           6/25 
  Running scriptlet: nvidia-driver-libs-3:525.85.12-1.el9.x86_64                                                                                           6/25 
  Installing       : nvidia-driver-cuda-libs-3:525.85.12-1.el9.x86_64                                                                                      7/25 
  Running scriptlet: nvidia-driver-cuda-libs-3:525.85.12-1.el9.x86_64                                                                                      7/25 
  Installing       : nvidia-driver-NvFBCOpenGL-3:525.85.12-1.el9.x86_64                                                                                    8/25 
  Running scriptlet: nvidia-driver-NvFBCOpenGL-3:525.85.12-1.el9.x86_64                                                                                    8/25 
  Installing       : xorg-x11-proto-devel-2021.4-2.el9.noarch                                                                                              9/25 
  Installing       : libXau-devel-1.0.9-8.el9.x86_64                                                                                                      10/25 
  Installing       : libxcb-devel-1.13.1-9.el9.x86_64                                                                                                     11/25 
  Installing       : libX11-devel-1.7.0-7.el9.x86_64                                                                                                      12/25 
  Installing       : nvidia-libXNVCtrl-devel-3:525.85.12-1.el9.x86_64                                                                                     13/25 
  Installing       : nvidia-driver-devel-3:525.85.12-1.el9.x86_64                                                                                         14/25 
  Installing       : ocl-icd-2.2.13-4.el9.x86_64                                                                                                          15/25 
  Installing       : opencl-filesystem-1.0-15.el9.noarch                                                                                                  16/25 
  Running scriptlet: nvidia-persistenced-3:525.85.12-1.el9.x86_64                                                                                         17/25 
  Installing       : nvidia-persistenced-3:525.85.12-1.el9.x86_64                                                                                         17/25 
  Running scriptlet: nvidia-persistenced-3:525.85.12-1.el9.x86_64                                                                                         17/25 
  Installing       : nvidia-driver-cuda-3:525.85.12-1.el9.x86_64                                                                                          18/25 
  Installing       : dnf-plugin-nvidia-2.0-1.el9.noarch                                                                                                   19/25 
  Installing       : nvidia-kmod-common-3:525.85.12-1.el9.noarch                                                                                          20/25 
  Running scriptlet: nvidia-kmod-common-3:525.85.12-1.el9.noarch                                                                                          20/25 
/sbin/grubby

  Installing       : nvidia-driver-3:525.85.12-1.el9.x86_64                                                                                               21/25 
  Running scriptlet: nvidia-driver-3:525.85.12-1.el9.x86_64                                                                                               21/25 
  Installing       : nvidia-modprobe-3:525.85.12-1.el9.x86_64                                                                                             22/25 
  Installing       : nvidia-settings-3:525.85.12-1.el9.x86_64                                                                                             23/25 
  Installing       : nvidia-xconfig-3:525.85.12-1.el9.x86_64                                                                                              24/25 
  Installing       : cuda-drivers-525.85.12-1.x86_64                                                                                                      25/25 
  Running scriptlet: cuda-drivers-525.85.12-1.x86_64                                                                                                      25/25 
  Verifying        : cuda-drivers-525.85.12-1.x86_64                                                                                                       1/25 
  Verifying        : dnf-plugin-nvidia-2.0-1.el9.noarch                                                                                                    2/25 
  Verifying        : kmod-nvidia-525.85.12-5.14.0-162.12.1-3:525.85.12-3.el9_1.x86_64                                                                      3/25 
  Verifying        : nvidia-driver-3:525.85.12-1.el9.x86_64                                                                                                4/25 
  Verifying        : nvidia-driver-NVML-3:525.85.12-1.el9.x86_64                                                                                           5/25 
  Verifying        : nvidia-driver-NvFBCOpenGL-3:525.85.12-1.el9.x86_64                                                                                    6/25 
  Verifying        : nvidia-driver-cuda-3:525.85.12-1.el9.x86_64                                                                                           7/25 
  Verifying        : nvidia-driver-cuda-libs-3:525.85.12-1.el9.x86_64                                                                                      8/25 
  Verifying        : nvidia-driver-devel-3:525.85.12-1.el9.x86_64                                                                                          9/25 
  Verifying        : nvidia-driver-libs-3:525.85.12-1.el9.x86_64                                                                                          10/25 
  Verifying        : nvidia-kmod-common-3:525.85.12-1.el9.noarch                                                                                          11/25 
  Verifying        : nvidia-libXNVCtrl-3:525.85.12-1.el9.x86_64                                                                                           12/25 
  Verifying        : nvidia-libXNVCtrl-devel-3:525.85.12-1.el9.x86_64                                                                                     13/25 
  Verifying        : nvidia-modprobe-3:525.85.12-1.el9.x86_64                                                                                             14/25 
  Verifying        : nvidia-persistenced-3:525.85.12-1.el9.x86_64                                                                                         15/25 
  Verifying        : nvidia-settings-3:525.85.12-1.el9.x86_64                                                                                             16/25 
  Verifying        : nvidia-xconfig-3:525.85.12-1.el9.x86_64                                                                                              17/25 
  Verifying        : xorg-x11-proto-devel-2021.4-2.el9.noarch                                                                                             18/25 
  Verifying        : egl-wayland-1.1.9-2.el9.x86_64                                                                                                       19/25 
  Verifying        : opencl-filesystem-1.0-15.el9.noarch                                                                                                  20/25 
  Verifying        : libXau-devel-1.0.9-8.el9.x86_64                                                                                                      21/25 
  Verifying        : libxcb-devel-1.13.1-9.el9.x86_64                                                                                                     22/25 
  Verifying        : libvdpau-1.4-6.el9.x86_64                                                                                                            23/25 
  Verifying        : ocl-icd-2.2.13-4.el9.x86_64                                                                                                          24/25 
  Verifying        : libX11-devel-1.7.0-7.el9.x86_64                                                                                                      25/25 

Installed:
  cuda-drivers-525.85.12-1.x86_64                                         dnf-plugin-nvidia-2.0-1.el9.noarch                                                   
  egl-wayland-1.1.9-2.el9.x86_64                                          kmod-nvidia-525.85.12-5.14.0-162.12.1-3:525.85.12-3.el9_1.x86_64                     
  libX11-devel-1.7.0-7.el9.x86_64                                         libXau-devel-1.0.9-8.el9.x86_64                                                      
  libvdpau-1.4-6.el9.x86_64                                               libxcb-devel-1.13.1-9.el9.x86_64                                                     
  nvidia-driver-3:525.85.12-1.el9.x86_64                                  nvidia-driver-NVML-3:525.85.12-1.el9.x86_64                                          
  nvidia-driver-NvFBCOpenGL-3:525.85.12-1.el9.x86_64                      nvidia-driver-cuda-3:525.85.12-1.el9.x86_64                                          
  nvidia-driver-cuda-libs-3:525.85.12-1.el9.x86_64                        nvidia-driver-devel-3:525.85.12-1.el9.x86_64                                         
  nvidia-driver-libs-3:525.85.12-1.el9.x86_64                             nvidia-kmod-common-3:525.85.12-1.el9.noarch                                          
  nvidia-libXNVCtrl-3:525.85.12-1.el9.x86_64                              nvidia-libXNVCtrl-devel-3:525.85.12-1.el9.x86_64                                     
  nvidia-modprobe-3:525.85.12-1.el9.x86_64                                nvidia-persistenced-3:525.85.12-1.el9.x86_64                                         
  nvidia-settings-3:525.85.12-1.el9.x86_64                                nvidia-xconfig-3:525.85.12-1.el9.x86_64                                              
  ocl-icd-2.2.13-4.el9.x86_64                                             opencl-filesystem-1.0-15.el9.noarch                                                  
  xorg-x11-proto-devel-2021.4-2.el9.noarch                               

Complete!
#

I’ll try DKMS as well to verify it does the same thing as well. I added the vault repo:

sudo dnf config-manager --add-repo  https://download.rockylinux.org/vault/rocky/9.0/BaseOS/x86_64/os/
# sudo dnf info --showduplicates kernel-core | grep -e Version -e Release | awk '{print $NF}' | paste -sd '-\n'
5.14.0-162.12.1.el9_1.0.1
5.14.0-162.12.1.el9_1.0.2
5.14.0-162.6.1.el9_1
5.14.0-162.12.1.el9_1.0.2
5.14.0-70.30.1.el9_0

Using latest kernel with dkms:

# uname -a 
Linux hostname.fqdn.edu 5.14.0-162.12.1.el9_1.0.2.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 30 22:14:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
# dnf module install nvidia-driver:latest-dkms
Last metadata expiration check: 0:16:21 ago on Mon 06 Feb 2023 09:49:02 AM CST.
Dependencies resolved.
================================================================================================================================================================
 Package                                    Architecture              Version                                        Repository                            Size
================================================================================================================================================================
Installing group/module packages:
 cuda-drivers                               x86_64                    525.85.12-1                                    cuda-rhel9-x86_64                    8.1 k
 kmod-nvidia-latest-dkms                    x86_64                    3:525.85.12-1.el9                              cuda-rhel9-x86_64                     43 M
 nvidia-driver                              x86_64                    3:525.85.12-1.el9                              cuda-rhel9-x86_64                     43 M
 nvidia-kmod-common                         noarch                    3:525.85.12-1.el9                              cuda-rhel9-x86_64                     12 k
 nvidia-modprobe                            x86_64                    3:525.85.12-1.el9                              cuda-rhel9-x86_64                     28 k
 nvidia-settings                            x86_64                    3:525.85.12-1.el9                              cuda-rhel9-x86_64                    1.7 M
 nvidia-xconfig                             x86_64                    3:525.85.12-1.el9                              cuda-rhel9-x86_64                     96 k
Installing dependencies:
 dkms                                       noarch                    3.0.10-1.el9                                   epel                                  84 k
 dnf-plugin-nvidia                          noarch                    2.0-1.el9                                      cuda-rhel9-x86_64                     13 k
 kernel-devel-matched                       x86_64                    5.14.0-162.12.1.el9_1.0.2                      appstream                            1.9 M
Installing module profiles:
 nvidia-driver/default                                                                                                                                         
Enabling module streams:
 nvidia-driver                                                        latest-dkms                                                                              

Transaction Summary
================================================================================================================================================================
Install  10 Packages

Total download size: 89 M
Installed size: 154 M
Is this ok [y/N]: y
Downloading Packages:
(1/10): cuda-drivers-525.85.12-1.x86_64.rpm                                                                                      40 kB/s | 8.1 kB     00:00    
(2/10): dnf-plugin-nvidia-2.0-1.el9.noarch.rpm                                                                                   63 kB/s |  13 kB     00:00    
(3/10): nvidia-kmod-common-525.85.12-1.el9.noarch.rpm                                                                           175 kB/s |  12 kB     00:00    
(4/10): nvidia-modprobe-525.85.12-1.el9.x86_64.rpm                                                                              355 kB/s |  28 kB     00:00    
(5/10): nvidia-settings-525.85.12-1.el9.x86_64.rpm                                                                               10 MB/s | 1.7 MB     00:00    
(6/10): nvidia-xconfig-525.85.12-1.el9.x86_64.rpm                                                                               908 kB/s |  96 kB     00:00    
(7/10): dkms-3.0.10-1.el9.noarch.rpm                                                                                            180 kB/s |  84 kB     00:00    
(8/10): kernel-devel-matched-5.14.0-162.12.1.el9_1.0.2.x86_64.rpm                                                               6.8 MB/s | 1.9 MB     00:00    
(9/10): kmod-nvidia-latest-dkms-525.85.12-1.el9.x86_64.rpm                                                                       24 MB/s |  43 MB     00:01    
(10/10): nvidia-driver-525.85.12-1.el9.x86_64.rpm                                                                                23 MB/s |  43 MB     00:01    
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                            33 MB/s |  89 MB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: cuda-drivers-525.85.12-1.x86_64                                                                                                        1/1 
  Preparing        :                                                                                                                                        1/1 
  Installing       : kernel-devel-matched-5.14.0-162.12.1.el9_1.0.2.x86_64                                                                                 1/10 
  Installing       : dkms-3.0.10-1.el9.noarch                                                                                                              2/10 
  Running scriptlet: dkms-3.0.10-1.el9.noarch                                                                                                              2/10 
  Installing       : dnf-plugin-nvidia-2.0-1.el9.noarch                                                                                                    3/10 
  Installing       : kmod-nvidia-latest-dkms-3:525.85.12-1.el9.x86_64                                                                                      4/10 
  Running scriptlet: kmod-nvidia-latest-dkms-3:525.85.12-1.el9.x86_64                                                                                      4/10 
  Installing       : nvidia-kmod-common-3:525.85.12-1.el9.noarch                                                                                           5/10 
  Running scriptlet: nvidia-kmod-common-3:525.85.12-1.el9.noarch                                                                                           5/10 
/sbin/grubby

  Installing       : nvidia-driver-3:525.85.12-1.el9.x86_64                                                                                                6/10 
  Running scriptlet: nvidia-driver-3:525.85.12-1.el9.x86_64                                                                                                6/10 
  Installing       : nvidia-modprobe-3:525.85.12-1.el9.x86_64                                                                                              7/10 
  Installing       : nvidia-settings-3:525.85.12-1.el9.x86_64                                                                                              8/10 
  Installing       : nvidia-xconfig-3:525.85.12-1.el9.x86_64                                                                                               9/10 
  Installing       : cuda-drivers-525.85.12-1.x86_64                                                                                                      10/10 
  Running scriptlet: cuda-drivers-525.85.12-1.x86_64                                                                                                      10/10 
  Verifying        : cuda-drivers-525.85.12-1.x86_64                                                                                                       1/10 
  Verifying        : dnf-plugin-nvidia-2.0-1.el9.noarch                                                                                                    2/10 
  Verifying        : kmod-nvidia-latest-dkms-3:525.85.12-1.el9.x86_64                                                                                      3/10 
  Verifying        : nvidia-driver-3:525.85.12-1.el9.x86_64                                                                                                4/10 
  Verifying        : nvidia-kmod-common-3:525.85.12-1.el9.noarch                                                                                           5/10 
  Verifying        : nvidia-modprobe-3:525.85.12-1.el9.x86_64                                                                                              6/10 
  Verifying        : nvidia-settings-3:525.85.12-1.el9.x86_64                                                                                              7/10 
  Verifying        : nvidia-xconfig-3:525.85.12-1.el9.x86_64                                                                                               8/10 
  Verifying        : dkms-3.0.10-1.el9.noarch                                                                                                              9/10 
  Verifying        : kernel-devel-matched-5.14.0-162.12.1.el9_1.0.2.x86_64                                                                                10/10 

Installed:
  cuda-drivers-525.85.12-1.x86_64                            dkms-3.0.10-1.el9.noarch                              dnf-plugin-nvidia-2.0-1.el9.noarch           
  kernel-devel-matched-5.14.0-162.12.1.el9_1.0.2.x86_64      kmod-nvidia-latest-dkms-3:525.85.12-1.el9.x86_64      nvidia-driver-3:525.85.12-1.el9.x86_64       
  nvidia-kmod-common-3:525.85.12-1.el9.noarch                nvidia-modprobe-3:525.85.12-1.el9.x86_64              nvidia-settings-3:525.85.12-1.el9.x86_64     
  nvidia-xconfig-3:525.85.12-1.el9.x86_64                   

Complete!

With secureboot on, UEFI and luks disk encryption after reboot with latest-dkms module:

# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Attached nvidia-bug-report.sh
nvidia-bug-report.log.gz (89.8 KB)

What would be the best way to install this old kernel to try it?

Name         : kernel-core
Version      : 5.14.0
Release      : 70.30.1.el9_0
Architecture : x86_64
Size         : 34 M
Source       : kernel-5.14.0-70.30.1.el9_0.src.rpm
Repository   : download.rockylinux.org_vault_rocky_9.0_BaseOS_x86_64_
Summary      : The Linux kernel
URL          : https://www.kernel.org/
License      : GPLv2 and Redistributable, no modification permitted
Description  : The kernel package contains the Linux kernel (vmlinuz), the core of any
             : Linux operating system.  The kernel handles the basic functions
             : of the operating system: memory allocation, process allocation, device
             : input and output, etc.