Notice: CUDA Linux Repository Key Rotation

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install packages from CUDA repositories.

To ensure continued access to the latest NVIDIA software, please complete the following steps:

Remove Outdated Signing Key

Debian, Ubuntu, WSL

$ sudo apt-key del 7fa2af80

Fedora, RHEL, openSUSE, SLES

$ sudo rpm --erase gpg-pubkey-7fa2af80*

Install New Key

For Debian-based distributions, including Ubuntu, you must additionally complete the following steps:

Install New cuda-keyring Package

To avoid the need for manual key installation steps, NVIDIA is providing a new helper package that will automate the installation of new signing keys for the NVIDIA repositories.

Replace $distro/$arch in the following commands with values appropriate for your OS; for example:

  • debian10/x86_64
  • debian11/x86_64
  • ubuntu1604/x86_64
  • ubuntu1804/cross-linux-sbsa
  • ubuntu1804/ppc64el
  • ubuntu1804/sbsa
  • ubuntu1804/x86_64
  • ubuntu2004/cross-linux-sbsa
  • ubuntu2004/sbsa
  • ubuntu2004/x86_64
  • ubuntu2204/sbsa
  • ubuntu2204/x86_64
  • wsl-ubuntu/x86_64

Debian, Ubuntu, WSL

$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb

$ sudo dpkg -i cuda-keyring_1.0-1_all.deb

Alternate method: Manually Install New Signing Key

If you are unable to install the cuda-keyring package, you can optionally install the new signing key manually (although this is not the recommended method).

Debian, Ubuntu, WSL

$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub

RPM distros

On fresh installation Fedora, RHEL, openSUSE, or SLES as dnf/yum/zypper will prompt the user to accept new keys when the repository signing key changes. Indicate you accept the change when prompted.

Replace $distro/$arch in the following commands with values appropriate for your OS; for example:

  • fedora32/x86_64
  • fedora33/x86_64
  • fedora34/x86_64
  • fedora35/x86_64
  • opensuse15/x86_64
  • rhel7/ppc64le
  • rhel7/x86_64
  • rhel8/cross-linux-sbsa
  • rhel8/ppc64le
  • rhel8/sbsa
  • rhel8/x86_64
  • sles15/cross-linux-sbsa
  • sles15/sbsa
  • sles15/x86_64

For upgrades, on RPM-based distros including Fedora, RHEL, SUSE, you must additionally complete the following steps:

Fedora and RHEL 8

$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

RHEL 7

$ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/$arch/cuda-rhel7.repo

openSUSe and SLES

$ sudo zypper removerepo cuda-$distro-$arch

$ sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

Working with Containers

CUDA applications built using older NGC base containers may contain outdated repository keys. If you build Docker containers using these images as a base and update the package manager or install additional NVIDIA packages as part of your Dockerfile, these commands may fail as they would on a non-container system. To work around this, integrate the above commands into the Dockerfile you use to build the container.

Existing containers in which the package manager is not used to install updates will not be affected by this key rotation.

Working with the NVIDIA GPU Operator

Current users of the GPU Operator on Ubuntu distributions may be affected by the rotation of the CUDA GPG keys, where some of the containers managed by the GPU Operator may fail to start with the following error:

Stopping NVIDIA persistence daemon... Unloading NVIDIA driver kernel modules... Unmounting NVIDIA driver rootfs... Checking NVIDIA driver packages... Updating the package cache... W: GPG error: [https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64](https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.download.nvidia.com%2Fcompute%2Fcuda%2Frepos%2Fubuntu2004%2Fx86_64&data=05%7C01%7Croarmstrong%40nvidia.com%7C41163a09f38040f818e008da29f754a3%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637868439487650977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=K%2Fu6Kkzu5CLt5EN7wHurqtFtRc9ZiQePhD%2BweXHCOrc%3D&reserved=0) InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is no longer signed.

NVIDIA will be publishing new images for the driver containers (by overwriting existing image tags). Users can work around this error by updating the existing clusterPolicy to pull new images:

$ kubectl edit clusterpolicy

and set driver.imagePullPolicy=Always

This step will result in the GPU Operator pulling the updated images. Note that new installations of the GPU Operator should be unaffected by this change and do not require any clusterPolicy updates. Users of the GPU Operator on RHEL or OpenShift are also not affected by this change.

3 Likes