We’ve not been using the mlnx_ofed stack. We’re required to mirror packages locally and ‘createrepo’ against that before provisioning downstream. All compute nodes on all HPC Asset clusters install the Infiniband Support package group. Through last week this did not pull in ‘mft’ as a dependency because the mlnx_ofed repo was not enabled. This week we’re trying to add knem, kmod-knem, and ucx-knem to the compute image, so we enabled the NVidia OFED repodef for the clients; this points to a cluster-local createrepo’d mirror of {new_users_can’t_post_multiple_links}://linux.mellanox.com/public/repo/mlnx_ofed/latest/rhel8.7/ with the clients’ repodef referencing {releasever} and {basearch} as usual.
Enabling dnf searching into the mlnx_ofed stack caused the Infiniband Support package group install to fail with no-digest for the NVidia arm64 mft package. We can easily verify this on an aarch64 system but, if we try to verify it on an x86_64 system against the same package files, the digests are OK.
Why are the SHA256 and MD5 digests of a static aarch64 package file visible on an x86_64 box but not visible, for the same static file, on an aarch64 box running the same base os?
Here’s a chroot session into the image being built on an aarch64 system, starting with the minimal bits required to support chroot and dnf:
bash-4.4# dnf groupinstall 'Infiniband Support'
Last metadata expiration check: 1:07:49 ago on Fri Feb 24 20:16:21 2023.
No match for group package "libibmad"
Dependencies resolved.
Problem: cannot install both libibverbs-41.0-1.el8.aarch64 and libibverbs-59mlnx44-1.59056.aarch64
- package perftest-4.5-12.el8.aarch64 requires libefa.so.1()(64bit), but none of the providers can be installed
- package perftest-4.5-12.el8.aarch64 requires libefa.so.1(EFA_1.1)(64bit), but none of the providers can be installed
- cannot install the best candidate for the job
======================================================================================================================================================================================================
Package Architecture Version Repository Size
======================================================================================================================================================================================================
Installing:
mlnx-ofed-all noarch 5.9-0.5.6.0.rhel8.7 nvidia-mlnx-ofed 11 k
Installing group/module packages:
ibacm aarch64 59mlnx44-1.59056 nvidia-mlnx-ofed 84 k
<...snipsnip...>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 56 MB/s | 123 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
package mft-4.23.0-104.aarch64 does not verify: no digest
Here’s the manual download and digest check, on the same aarch64 system and in the same chroot session, showing inability to see the SHA256 and MD5 digests of the mft package but ability to see them in (arbitrarily chosen) perl-Digest-SHA:
bash-4.4# uname -rm
4.18.0-425.10.1.el8_7.aarch64 aarch64
bash-4.4# dnf clean packages
0 files removed
bash-4.4# dnf download mft
Last metadata expiration check: 1:28:21 ago on Fri Feb 24 20:16:21 2023.
mft-4.23.0-104.arm64.rpm 57 MB/s | 34 MB 00:00
bash-4.4# rpm -Kvv --nosignature mft-4.23.0-104.arm64.rpm 2>&1 |grep -i digest
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
MD5 digest: NOTFOUND
bash-4.4# dnf clean packages
0 files removed
bash-4.4# dnf download perl-Digest-SHA
Last metadata expiration check: 1:28:53 ago on Fri Feb 24 20:16:21 2023.
perl-Digest-SHA-6.02-1.el8.aarch64.rpm 19 MB/s | 63 kB 00:00
bash-4.4# rpm -Kvv --nosignature perl-Digest-SHA-6.02-1.el8.aarch64.rpm 2>&1 |grep -i digest
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
perl-Digest-SHA-6.02-1.el8.aarch64.rpm:
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
In case the problem is in the [as of yet minimal] system image or chroot session, here’s the same check on the same system, outside of the chroot and new system image, accessing the same package files:
16:49:21 root@compute104.godzilla:/var/image/godzilla/20230214-compute.aarch64.el8/rootfs/root # uname -rm
4.18.0-425.10.1.el8_7.aarch64 aarch64
16:49:23 root@compute104.godzilla:/var/image/godzilla/20230214-compute.aarch64.el8/rootfs/root # rpm -Kvv --nosignature mft-4.23.0-104.arm64.rpm 2>&1 |grep -i digest
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
MD5 digest: NOTFOUND
16:49:39 root@compute104.godzilla:/var/image/godzilla/20230214-compute.aarch64.el8/rootfs/root # rpm -Kvv --nosignature perl-Digest-SHA-6.02-1.el8.aarch64.rpm 2>&1 |grep -i digest
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
perl-Digest-SHA-6.02-1.el8.aarch64.rpm:
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
And the screwy part: here’s the same check done with the same files on the x86_64 infrastructure node responsible for serving the local package repo (built from a mirror of Index of /public/repo/mlnx_ofed/latest/rhel8.7 with ‘createrepo’ run against it). These are the same package files as before, for aarch64, but with the check run on an x86_64 system:
16:52:17 root@admin.godzilla:/var/repo/nvidia/public/repo/mlnx_ofed/latest/rhel8/aarch64 # uname -rm
4.18.0-425.10.1.el8_7.x86_64 x86_64
16:52:18 root@admin.godzilla:/var/repo/nvidia/public/repo/mlnx_ofed/latest/rhel8/aarch64 # rpm -Kvv --nosignature mft-4.23.0-104.arm64.rpm 2>&1 |grep -i digest
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
MD5 digest: OK
16:53:41 root@admin.godzilla:/var/repo/rocky-linux/8/AppStream/aarch64/os/Packages/p # rpm -Kvv --nosignature perl-Digest-SHA-6.02-1.el8.aarch64.rpm 2>&1 |grep -i digest
Header SHA1 digest: OK
Header SHA1 digest: OK
Header SHA1 digest: OK
perl-Digest-SHA-6.02-1.el8.aarch64.rpm:
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
MD5 digest: OK