When enabling FIPS mode in RHEL8, md5sums are disallowed due to FIPS restrictions, so CUDA rpms cannot be installed via yum/dnf
For example:
yum install libnpp-11-8
fails with:
Error: Transaction test error:
package libnpp-11-8-11.8.0.86-1.x86_64 does not verify: no digest
This is due to the file digests in md5 format. All official RHEL8 packages are built with SHA256 digests
rpm --checksig libnpp-11-8-11.8.0.86-1.x86_64.rpm
libnpp-11-8-11.8.0.86-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Header SHA1 digest: OK
V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
MD5 digest: OK
Grafana packages had the same issue, but seems they have solved it recently:
rpm --checksig -v grafana-9.1.7-1.x86_64.rpm
grafana-9.1.7-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID 24098cb6: NOKEY
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
V4 RSA/SHA512 Signature, key ID 24098cb6: NOKEY
MD5 digest: OK
Here is some info about sha256 digests when building RPMs
Adding SHA256 Digests to RPMs — Star Lab Software (Can’t post more than one link)
It would be great if NVIDIA could build packages with proper SHA256 digests
2 Likes
Agree with OP - it seems the packages available in the NVIDIA CUDA for EL8 repo are inconsistently built; some have more secure SHA256 digests, some without.
Good package that installs with no issue:
$ rpm -K -v cuda-drivers-470.161.03-1.x86_64.rpm
cuda-drivers-470.161.03-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: OK
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
V4 RSA/SHA512 Signature, key ID d42d0685: OK
Bad package with install failure and “no digest” error:
$ rpm -K -v nsight-compute-2020.2.1-2020.2.1.8-1.x86_64.rpm
nsight-compute-2020.2.1-2020.2.1.8-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: OK
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
V4 RSA/SHA512 Signature, key ID d42d0685: OK
MD5 digest: NOTFOUND
I’ve tried to open a support case with NVIDIA, but so far they haven’t been willing to address this.
System details:
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.7 (Ootpa)
$ uname -r
4.18.0-425.3.1.el8.x86_64
$ fips-mode-setup --check
FIPS mode is enabled.
1 Like
Greetings, I just wanted to +1 this request. Our organization uses Nvidia hardware and also the CUDA Development Toolkit (including nsight-compute) for scientific computing. However, we have stringent security requirements which mandate the use of good-quality (SHA-256) RPM payload digest algorithms.
Same problem here (NASA). And for the record at the moment it’s nsight-systems stopping it, and I’m pretty sure I have been able to install/update recently with FIPS enabled. So I’m not sure what’s going on.
[root@ts-5061834 ~]# rpm -K -vvv /var/cache/dnf/cuda-rhel8-x86_64-f1d7a46f058da57c/packages/cuda-nsight-systems-12-2-12.2.0-1.x86_64.rpm
ufdio: 1 reads, 17154 total bytes in 0.000005 secs
ufdio: 1 reads, 5442 total bytes in 0.000003 secs
ufdio: 1 reads, 17154 total bytes in 0.000008 secs
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/.key
D: couldn’t find any keys in /var/lib/rpm/pubkeys/.key
D: loading keyring from rpmdb
D: opening db environment /var/lib/rpm cdb:0x401
D: opening db index /var/lib/rpm/Packages 0x400 mode=0x0
D: locked db index /var/lib/rpm/Packages
D: opening db index /var/lib/rpm/Name 0x400 mode=0x0
D: read h# 1532
Header SHA1 digest: OK
D: added key gpg-pubkey-fd431d51-4ae0493b to keyring
D: read h# 1533
Header SHA1 digest: OK
D: added key gpg-pubkey-d4082792-5b32db75 to keyring
D: added subkey 0 of main key gpg-pubkey-d4082792-5b32db75 to keyring
D: read h# 2045
Header SHA1 digest: OK
D: added key gpg-pubkey-baadae52-49beffa4 to keyring
D: read h# 2048
Header SHA1 digest: OK
D: added key gpg-pubkey-2f86d6a1-5cf7cefb to keyring
D: read h# 2058
Header SHA1 digest: OK
D: added key gpg-pubkey-ab73fe5d-5a301712 to keyring
D: added subkey 0 of main key gpg-pubkey-ab73fe5d-5a301712 to keyring
D: read h# 2066
Header SHA1 digest: OK
D: added key gpg-pubkey-d42d0685-62589a51 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
/var/cache/dnf/cuda-rhel8-x86_64-f1d7a46f058da57c/packages/cuda-nsight-systems-12-2-12.2.0-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: OK
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
V4 RSA/SHA512 Signature, key ID d42d0685: OK
ufdio: 8 reads, 9669 total bytes in 0.000016 secs
D: closed db index /var/lib/rpm/Packages
D: closed db index /var/lib/rpm/Name
D: closed db environment /var/lib/rpm
I was getting nsight-systems failing, but now it seems to be all nsight packages.
This is just unacceptable.
RHEL8** FIPS mode is a hard, kernel block, it won’t even allow the function calls (so it looks like NOTFOUND), and nVidia – in 2022+ – is still using MD5/SHA1 for digests. nVidia really need to start using at least SHA256 digests. It’s largely CUDA 11 packages, but I still see it on earlier CUDA 12 packages too. Some of us take the time to deal with MoK/SecureBoot and module signatures too.
(Non-FIPS Mode – only last is proper)
[x86_64]$ rpm -Kv cuda-11-8-11.8.0-1.x86_64.rpm
cuda-11-8-11.8.0-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: OK
Header SHA1 digest: OK
V4 RSA/SHA512 Signature, key ID d42d0685: OK
MD5 digest: OK
[x86_64]$ rpm -Kv cuda-12-0-12.0.1-1.x86_64.rpm
cuda-12-0-12.0.1-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: OK
Header SHA1 digest: OK
V4 RSA/SHA512 Signature, key ID d42d0685: OK
MD5 digest: OK
[x86_64]$ rpm -Kv cuda-12-2-12.2.1-1.x86_64.rpm
cuda-12-2-12.2.1-1.x86_64.rpm:
Header V4 RSA/SHA512 Signature, key ID d42d0685: OK
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
V4 RSA/SHA512 Signature, key ID d42d0685: OK
MD5 digest: OK
**Red Hat released Update 6 of RHEL 7 (7.6) with the updated FIPS mode removal of SHA1, and ended up having to back it out, because it broke a lot of IHV/ISV repositories, among other things (possibly even 389/RHDS too?). But that was years ago, especially since RHEL7 came out in 2014. RHEL8 came out in 2019, so all major IHV/ISV support has been in the 2020s. It’s time to get with this decade nVidia.
1 Like