I run sudo apt update && sudo apt full-upgrade -y every day. Today it failed for an CUDA related reason that I think is related to this key rotation. Could anyone guide me on resolving this issue? I can’t update or install new packages now.
I followed the steps in the blog:
sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
Every time I run sudo apt-key del 7fa2af80 I get this output:
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
I’m concerned that I can’t update due to apt-key deprecation or that I haven’t deleted and updated they older key fast enough. I’m also concerned the problem with the MergeList is due to something at the file /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2204_x86%5f64_Packages and wonder if there is a way I can reset that file somehow.
$ sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:6 https://repo.nordvpn.com/deb/nordvpn/debian stable InRelease
Hit:7 https://ppa.launchpadcontent.net/yubico/stable/ubuntu jammy InRelease
Reading package lists... Error!
W: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2204_x86%5f64_Packages
E: The package lists or status file could not be parsed or opened.
Hi @MicahParks
Thank you for bring this to my attention. The default .deb compression changed from XZ to Zstandard in Ubuntu 22.04, which is not recognized by the build of the dpkg executable currently in use for updating the repository metadata.
We had four postings yesterday, one of which was for NCCL; these .deb packages are compressed with Zstandard, other packages in the repository continue use XZ compression.
Working to resolve this issue on our end, though it may also require users to delete /var/lib/apt/lists/*cuda_repos* after the repository metadata has been re-generated.
@kmittman Thank you for your response. Could you link the other postings so I can follow along for updates? My plan is to wait for a fix since I don’t want to change anything now and require more work when an official fix is available. In the meantime, I can’t update apt or install new packages, which is limiting during development, so I’ll want to be in the loop for any official fixes.
Hi @steven.ramboer and @xkszltl
I have filed an internal task to notify the cuDNN, NCCL, and TensorRT teams about this request for Ubuntu 22.04 .deb packages to be available (hopefully sooner than later). Also will ask about .deb packages for the Debian 11 repo too.
I’ve discovered this discussion while trying to update some CentOS 7 severs.
I have tried to follow the instructions here but they simply do not work. There is some circular logic happening and I have no idea how to break it:
I remove the bad key using sudo rpm -e gpg-pubkey-7fa2af80-576db785
I verify that it is gone with rpm -q gpg-pubkey --qf ‘%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n’
Yet when I try to install the latest nvidia driver, the bad key is RE-DOWNLOADED!
=============================================================================================================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================================================================================================
Updating:
nvidia-driver-latest-dkms x86_64 3:515.48.07-1.el7 cuda 23 M
Installing for dependencies:
egl-wayland x86_64 1.1.6-1.el7 epel 29 k
Updating for dependencies:
kmod-nvidia-latest-dkms x86_64 3:515.48.07-1.el7 cuda 30 M
nvidia-driver-latest-dkms-NVML x86_64 3:515.48.07-1.el7 cuda 468 k
nvidia-driver-latest-dkms-NvFBCOpenGL x86_64 3:515.48.07-1.el7 cuda 59 k
nvidia-driver-latest-dkms-cuda x86_64 3:515.48.07-1.el7 cuda 291 k
nvidia-driver-latest-dkms-cuda-libs x86_64 3:515.48.07-1.el7 cuda 54 M
nvidia-driver-latest-dkms-devel x86_64 3:515.48.07-1.el7 cuda 19 k
nvidia-driver-latest-dkms-libs x86_64 3:515.48.07-1.el7 cuda 177 M
nvidia-modprobe-latest-dkms x86_64 3:515.48.07-1.el7 cuda 34 k
nvidia-persistenced-latest-dkms x86_64 3:515.48.07-1.el7 cuda 36 k
nvidia-xconfig-latest-dkms x86_64 3:515.48.07-1.el7 cuda 95 k
Transaction Summary
=============================================================================================================================================================================================================================================================
Install ( 1 Dependent package)
Upgrade 1 Package (+10 Dependent packages)
Total size: 286 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda/packages/nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Importing GPG key 0x7FA2AF80:
Userid : "cudatools <cudatools@nvidia.com>"
Fingerprint: ae09 fe4b bd22 3a84 b2cc fce3 f60f 4b3d 7fa2 af80
From : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Is this ok [y/N]: y
Public key for nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64.rpm is not installed
Failing package is: 3:nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64
GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
How can I get the correct key? RPM is supposed to download it according to the documentation yet it’s getting the old key. Please help!
Hi @lh2332
Yes, you will need to update the .repo file for RHEL-based, Fedora, and SUSE distros. The cuda-rhel7.repo file on your system is pointing at the old GPG key location, which is why it is re-downloading it.
Install the new CUDA public GPG key: The new GPG public key for the CUDA repository (RPM-based distros) is d42d0685.On a fresh installation of RHEL, the yum package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted.
For upgrades, you must also also fetch an updated .repo entry:
Do you happen to have other cuda repos listed under /etc/yum.repos.d?
Yours called cuda in log, but these days it’s called cuda-rhel7, so probably you have an old version as well and that’s the one complaining.
The deprecated GPG pubkey does not match the RPMs in the repository …
# yum install libnvjpeg-11-0
[...]
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda-rhel7-x86_64/packages/libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Public key for libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm is not installed
libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm | 2.0 MB 00:00:00
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Importing GPG key 0x7FA2AF80:
Userid : "cudatools <cudatools@nvidia.com>"
Fingerprint: ae09 fe4b bd22 3a84 b2cc fce3 f60f 4b3d 7fa2 af80
From : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Is this ok [y/N]: y
Public key for libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm is not installed
Failing package is: libnvjpeg-11-0-11.1.1.245-1.x86_64
GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub