To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install packages from CUDA repositories.
To ensure continued access to the latest NVIDIA software, please complete the following steps:
Remove Outdated Signing Key
Debian, Ubuntu, WSL
$ sudo apt-key del 7fa2af80
Fedora, RHEL, openSUSE, SLES
$ sudo rpm --erase gpg-pubkey-7fa2af80*
Install New Key
For Debian-based distributions, including Ubuntu, you must additionally complete the following steps:
Install New cuda-keyring Package
To avoid the need for manual key installation steps, NVIDIA is providing a new helper package that will automate the installation of new signing keys for the NVIDIA repositories.
Replace $distro/$arch in the following commands with values appropriate for your OS; for example:
- debian10/x86_64
- debian11/x86_64
- ubuntu1604/x86_64
- ubuntu1804/cross-linux-sbsa
- ubuntu1804/ppc64el
- ubuntu1804/sbsa
- ubuntu1804/x86_64
- ubuntu2004/cross-linux-sbsa
- ubuntu2004/sbsa
- ubuntu2004/x86_64
- ubuntu2204/sbsa
- ubuntu2204/x86_64
- wsl-ubuntu/x86_64
Debian, Ubuntu, WSL
$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb
Alternate method: Manually Install New Signing Key
If you are unable to install the cuda-keyring package, you can optionally install the new signing key manually (although this is not the recommended method).
Debian, Ubuntu, WSL
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub
RPM distros
On fresh installation Fedora, RHEL, openSUSE, or SLES as dnf/yum/zypper will prompt the user to accept new keys when the repository signing key changes. Indicate you accept the change when prompted.
Replace $distro/$arch in the following commands with values appropriate for your OS; for example:
- fedora32/x86_64
- fedora33/x86_64
- fedora34/x86_64
- fedora35/x86_64
- opensuse15/x86_64
- rhel7/ppc64le
- rhel7/x86_64
- rhel8/cross-linux-sbsa
- rhel8/ppc64le
- rhel8/sbsa
- rhel8/x86_64
- sles15/cross-linux-sbsa
- sles15/sbsa
- sles15/x86_64
For upgrades, on RPM-based distros including Fedora, RHEL, SUSE, you must additionally complete the following steps:
Fedora and RHEL 8
$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo
RHEL 7
$ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/$arch/cuda-rhel7.repo
openSUSe and SLES
$ sudo zypper removerepo cuda-$distro-$arch
$ sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo
Working with Containers
CUDA applications built using older NGC base containers may contain outdated repository keys. If you build Docker containers using these images as a base and update the package manager or install additional NVIDIA packages as part of your Dockerfile, these commands may fail as they would on a non-container system. To work around this, integrate the above commands into the Dockerfile you use to build the container.
Existing containers in which the package manager is not used to install updates will not be affected by this key rotation.
Working with the NVIDIA GPU Operator
Current users of the GPU Operator on Ubuntu distributions may be affected by the rotation of the CUDA GPG keys, where some of the containers managed by the GPU Operator may fail to start with the following error:
Stopping NVIDIA persistence daemon... Unloading NVIDIA driver kernel modules... Unmounting NVIDIA driver rootfs... Checking NVIDIA driver packages... Updating the package cache... W: GPG error: [https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64](https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.download.nvidia.com%2Fcompute%2Fcuda%2Frepos%2Fubuntu2004%2Fx86_64&data=05%7C01%7Croarmstrong%40nvidia.com%7C41163a09f38040f818e008da29f754a3%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637868439487650977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=K%2Fu6Kkzu5CLt5EN7wHurqtFtRc9ZiQePhD%2BweXHCOrc%3D&reserved=0) InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is no longer signed.
NVIDIA will be publishing new images for the driver containers (by overwriting existing image tags). Users can work around this error by updating the existing clusterPolicy to pull new images:
$ kubectl edit clusterpolicy
and set driver.imagePullPolicy=Always
This step will result in the GPU Operator pulling the updated images. Note that new installations of the GPU Operator should be unaffected by this change and do not require any clusterPolicy updates. Users of the GPU Operator on RHEL or OpenShift are also not affected by this change.
Common Errors and Solutions on Debian-based distros
Duplicate .list entries
E: Conflicting values set for option Signed-By regarding source
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /:/usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.
Solution: If previously used add-apt-repository to enable the CUDA repository, then remove the duplicate entry
sudo sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list
Also check for and remove cuda*.list files under /etc/apt/sources.d/ directory.
New GPG key is not enrolled
Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY A4B46996 3BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is no longer signed.
Solution:
See above to install cuda-keyring package OR one of the manual enrollment methods for the 3bf863cc public key.
Machine Learning repository
W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used.
GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64
Release: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY F60F4B3D 7FA2AF80
Solution:
Remove the NVIDIA machine learning repository entry, it is no longer updated. Newer versions of cuDNN, NCCL, and TensorRT are available in the CUDA repository.
File has unexpected size
Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: XXX.XXX.XXX.XXX 443]
Hashes of expected file:
- Filesize:481481 [weak]
- SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
- SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
- MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
Solution: Report CDN issue to NVIDIA
