Notice: CUDA Linux Repository Key Rotation

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install packages from CUDA repositories.

To ensure continued access to the latest NVIDIA software, please complete the following steps:

Remove Outdated Signing Key

Debian, Ubuntu, WSL

$ sudo apt-key del 7fa2af80

Fedora, RHEL, openSUSE, SLES

$ sudo rpm --erase gpg-pubkey-7fa2af80*

Install New Key

For Debian-based distributions, including Ubuntu, you must additionally complete the following steps:

Install New cuda-keyring Package

To avoid the need for manual key installation steps, NVIDIA is providing a new helper package that will automate the installation of new signing keys for the NVIDIA repositories.

Replace $distro/$arch in the following commands with values appropriate for your OS; for example:

  • debian10/x86_64
  • debian11/x86_64
  • ubuntu1604/x86_64
  • ubuntu1804/cross-linux-sbsa
  • ubuntu1804/ppc64el
  • ubuntu1804/sbsa
  • ubuntu1804/x86_64
  • ubuntu2004/cross-linux-sbsa
  • ubuntu2004/sbsa
  • ubuntu2004/x86_64
  • ubuntu2204/sbsa
  • ubuntu2204/x86_64
  • wsl-ubuntu/x86_64

Debian, Ubuntu, WSL

$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb

$ sudo dpkg -i cuda-keyring_1.0-1_all.deb

Alternate method: Manually Install New Signing Key

If you are unable to install the cuda-keyring package, you can optionally install the new signing key manually (although this is not the recommended method).

Debian, Ubuntu, WSL

$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub

RPM distros

On fresh installation Fedora, RHEL, openSUSE, or SLES as dnf/yum/zypper will prompt the user to accept new keys when the repository signing key changes. Indicate you accept the change when prompted.

Replace $distro/$arch in the following commands with values appropriate for your OS; for example:

  • fedora32/x86_64
  • fedora33/x86_64
  • fedora34/x86_64
  • fedora35/x86_64
  • opensuse15/x86_64
  • rhel7/ppc64le
  • rhel7/x86_64
  • rhel8/cross-linux-sbsa
  • rhel8/ppc64le
  • rhel8/sbsa
  • rhel8/x86_64
  • sles15/cross-linux-sbsa
  • sles15/sbsa
  • sles15/x86_64

For upgrades, on RPM-based distros including Fedora, RHEL, SUSE, you must additionally complete the following steps:

Fedora and RHEL 8

$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

RHEL 7

$ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/$arch/cuda-rhel7.repo

openSUSe and SLES

$ sudo zypper removerepo cuda-$distro-$arch

$ sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

Working with Containers

CUDA applications built using older NGC base containers may contain outdated repository keys. If you build Docker containers using these images as a base and update the package manager or install additional NVIDIA packages as part of your Dockerfile, these commands may fail as they would on a non-container system. To work around this, integrate the above commands into the Dockerfile you use to build the container.

Existing containers in which the package manager is not used to install updates will not be affected by this key rotation.

Working with the NVIDIA GPU Operator

Current users of the GPU Operator on Ubuntu distributions may be affected by the rotation of the CUDA GPG keys, where some of the containers managed by the GPU Operator may fail to start with the following error:

Stopping NVIDIA persistence daemon... Unloading NVIDIA driver kernel modules... Unmounting NVIDIA driver rootfs... Checking NVIDIA driver packages... Updating the package cache... W: GPG error: [https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64](https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.download.nvidia.com%2Fcompute%2Fcuda%2Frepos%2Fubuntu2004%2Fx86_64&data=05%7C01%7Croarmstrong%40nvidia.com%7C41163a09f38040f818e008da29f754a3%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637868439487650977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=K%2Fu6Kkzu5CLt5EN7wHurqtFtRc9ZiQePhD%2BweXHCOrc%3D&reserved=0) InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is no longer signed.

NVIDIA will be publishing new images for the driver containers (by overwriting existing image tags). Users can work around this error by updating the existing clusterPolicy to pull new images:

$ kubectl edit clusterpolicy

and set driver.imagePullPolicy=Always

This step will result in the GPU Operator pulling the updated images. Note that new installations of the GPU Operator should be unaffected by this change and do not require any clusterPolicy updates. Users of the GPU Operator on RHEL or OpenShift are also not affected by this change.

Common Errors and Solutions on Debian-based distros

Duplicate .list entries

E: Conflicting values set for option Signed-By regarding source
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /:/usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.

Solution: If previously used add-apt-repository to enable the CUDA repository, then remove the duplicate entry

sudo sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list

Also check for and remove cuda*.list files under /etc/apt/sources.d/ directory.

New GPG key is not enrolled

Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY A4B46996 3BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is no longer signed.

Solution:

See above to install cuda-keyring package OR one of the manual enrollment methods for the 3bf863cc public key.

Machine Learning repository

W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used.
GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64
Release: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY F60F4B3D 7FA2AF80

Solution:

Remove the NVIDIA machine learning repository entry, it is no longer updated. Newer versions of cuDNN, NCCL, and TensorRT are available in the CUDA repository.

File has unexpected size

Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: XXX.XXX.XXX.XXX 443]
Hashes of expected file:
- Filesize:481481 [weak]
- SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
- SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
- MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Solution: Report CDN issue to NVIDIA

3 Likes

Hello, I have just tried to use your solution to solve the Signing key issues but, it crashes with the following error. A simple docker file has been attached to replicate the job.

Sending build context to Docker daemon  2.573GB
Step 1/3 : FROM nvcr.io/nvidia/cuda:11.6.2-devel-ubuntu20.04
---> d333415bf915
Step 2/3 : RUN apt-key del 7fa2af80     && apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
---> Running in bd59abbc71d9
OK
Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.P7ZYWwRuXi/gpg.1.sh --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub                                                                                     
gpg: requesting key from 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub'
gpg: key A4B469963BF863CC: public key "cudatools <cudatools@nvidia.com>" imported                                                                                                                                                            
gpg: Total number processed: 1                                                                                                                                                                                                               
gpg:               imported: 1                                                                                                                                                                                                               
Removing intermediate container bd59abbc71d9                                                                                                                                                                                                 
---> 4f3144e5c274
Step 3/3 : RUN apt-get update
---> Running in b4264c8b83c5
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1575 B]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [481 kB]
Err:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.199.20.126 443]
Hashes of expected file:
- Filesize:481481 [weak]
- SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
- SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
- MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1777 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1139 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [870 kB]
Get:12 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2192 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1216 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [30.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1155 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [26.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [51.2 kB]
Fetched 21.9 MB in 5s (4748 kB/s)
Reading package lists...
E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.199.20.126 443]
  Hashes of expected file:                                                                                                                                                                                                                  
   - Filesize:481481 [weak]                                                                                                                                                                                                                 
   - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b                                                                                                                                                                
   - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]                                                                                                                                                                                   
   - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]                                                                                                                                                                                         
  Release file created at: Mon, 25 Apr 2022 23:27:19 +0000                                                                                                                                                                                  
E: Some index files failed to download. They have been ignored, or old ones used instead.                                                                                                                                                    
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

Could you please let me know how to overcome this issue? Thanks
Dockerfile (219 Bytes)

2 Likes

thy this

RUN apt-key del 7fa2af80
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb

RUN dpkg -i cuda-keyring_1.0-1_all.deb

RUN sed -i 's/deb https:\/\/developer.download.nvidia.com\/compute\/cuda\/repos\/ubuntu2004\/x86_64.*//g' /etc/apt/sources.list

I believe wget can only run after apt-get update, is it right? or there is a way to install wget before?

The recommended approach doesn’t work (it creates duplicated apt source). The alternate approach does. However, for a new cuda image, we also need to do the same for machine-learning repository. So this would work for a 18.04 cuda 10.2 image

Can’t copy paste the text because the forum doesn’t allow me to post reply with nvidia host (sigh…)

1 Like

if you don’t have wget of curl :
instead of curl or wget you can download in the same folder as you dockerfile download the file from

# on the local machine before build download this file
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb

# then in dockerfile

RUN apt-key del 7fa2af80

COPY cuda-keyring_1.0-1_all.deb cuda-keyring_1.0-1_all.deb

RUN dpkg -i cuda-keyring_1.0-1_all.deb

RUN sed -i 's/deb https:\/\/developer.download.nvidia.com\/compute\/cuda\/repos\/ubuntu2004\/x86_64.*//g' /etc/apt/sources.list

1 Like

you can paste url using then “insert code” snippet

it seems to work for me

Is there a recommended action for users of the Kubernetes gpu-operator? I’m still digging down the rabbit hole but I’m not sure where to insert the commands to change the keys.

RUN sh -c 'echo "APT { Get { AllowUnauthenticated \"1\"; }; };" > /etc/apt/apt.conf.d/99allow_unauth'

RUN apt -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true update
RUN apt-get install -y curl wget

RUN apt-key del 7fa2af80
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb
RUN rm -f /etc/apt/sources.list.d/cuda.list /etc/apt/apt.conf.d/99allow_unauth cuda-keyring_1.0-1_all.deb

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A4B469963BF863CC F60F4B3D7FA2AF80

RUN apt-get update && apt-get upgrade -y
4 Likes

This solution works for me thanks

it looks like it worked for me.
here is the code

apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

add it before

apt-get update

Is there a mailing list we can subscribe to that would have announcements like this in it? I’m having trouble finding anything that exists (that is not tied to a single user account). Would love to ensure our team here gets notifications as early as possible.

1 Like

This works fine for me

apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub

But this doesn’t work

apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

I need to use the machine-learning repo but the public key is invalid now.

The wget command for cuda-keyring (https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb) is giving 404 not found error.

Are you sure that $distro/$arch is being replaced as indicated in the repo?

@akashsjhs I think @Dgromov is right, you need to replace $distro/$arch in your command line with the appropriate values for your system from the list in the original post.

–2022-05-05 17:54:48-- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)… 152.199.39.144
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|152.199.39.144|:443… connected.
HTTP request sent, awaiting response… 404 Not Found
2022-05-05 17:54:48 ERROR 404: Not Found.

is the site down?

@thaha1 We do not yet support Ubuntu 22.04 in CUDA and there is no corresponding repository.

1 Like