Updating the CUDA Linux GPG Repository Key

@frenzi This does solve it for me as well. How did you find out?

@roarmstrong @kmittman I often pull official docker image from https://hub.docker.com/r/nvidia/cuda.
Do you have a plan to replace these docker images with updated GPG Key?
Because, I don’t know if the following method is a temporary workaround.
https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

@dandelion1124 The containers are being updated with the new signing keys and are being published as we speak, so this is a temporary requirement. The container images for 11.0+ should all be live at this point.

@roarmstrong Thank you for your reply.

The containers are being updated with the new signing keys and are being published as we speak, so this is a temporary requirement. The container images for 11.0+ should all be live at this point.

I checked it.

docker run -it --rm --gpus all nvidia/cuda:11.0-base bash

In this container, I executed apt update. But, it occurs GPG error.

apt update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1575 B]
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease               
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]                         
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]               
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]               
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1777 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]            
Get:10 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1139 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]           
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1216 kB]
Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [870 kB]      
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [30.3 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2192 kB]         
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1155 kB]   
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [51.2 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [26.0 kB]
Reading package lists... Done                               
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Please tell me your update schedule.

@dandelion1124 The nvidia/cuda:11.0-base tag is outdated, can you please use 11.0.3-base-ubuntu20.04 instead?

@roarmstrong

The nvidia/cuda:11.0-base tag is outdated, can you please use 11.0.3-base-ubuntu20.04 instead?

I tried it. As a result, there is no problem. Thanks.

docker run -it --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 bash
apt update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1575 B]
Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [481 kB]       
Get:3 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]                                     
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1777 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1139 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [870 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]               
Get:12 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]               
Get:14 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2192 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1216 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1155 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [30.3 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [26.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [51.2 kB]
Fetched 22.4 MB in 6s (3883 kB/s)                             
Reading package lists... Done
Building dependency tree       
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

The nvidia/cuda:11.0-base tag is outdated,

The reason I used nvidia/cuda:11.0-base is because your comment has the following explanation.

The container images for 11.0+ should all be live at this point.

So, I misunderstood that nvidia/cuda:11.0-base was updated with new signing keys.

I was able to fix nvcr.io/nvidia/deepstream:5.1-21.02-base with

RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub

Can I get confirmation that this is a temporary fix and updated versions of all containers are on the way? When can we expect them to be live on NGC?

Nvidia have not actually updated their old images (tags) with the new keys, so pulling the same image again isn’t going to solve this problem. If you look at ngc, the last time that 470.82.01-ubuntu20.04 was updated was 01/20/2022, before this issue, i.e. the key changes. Since the rest of my cluster is running the aformentioned version that comes with whatever helm release I used, I decided to use the 470-signed-ubuntu20.04 image so that the driver major versions match.

Hi @someuser - I have the exact same problem and errors as you, however I haven’t been able to work around it yet. Could you please help with what exactly were your commands in the Dockerfile? This is what I have right now:

RUN sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list.d/* \
         && sed -i '/developer\.download\.nvidia\.com\/compute\/machine-learning\/repos/d' /etc/apt/sources.list.d/* \
         && apt-key del 7fa2af80 \
         && wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb \
         && dpkg -i cuda-keyring_1.0-1_all.deb

RUN apt-get update -y \
         && apt-get install -y --no-install-recommends \
...

Any suggestions will be of great help - thank you!

Hi @pragya.jswl
indeed I had to refine a bit, as apparently some layers still were in my local cache (and then some commands still silently failed in between). Here is what I have, and that works ok for me with docker build --no-cache:

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04

ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

# update CUDA keyring
RUN sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list.d/* && \
    sed -i '/developer\.download\.nvidia\.com\/compute\/machine-learning\/repos/d' /etc/apt/sources.list.d/* && \
    apt-key del 7fa2af80 
RUN apt-get update && apt-get install wget 
RUN cd /tmp && \
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \
    dpkg -i cuda-keyring_1.0-1_all.deb && \
    apt-get update

Hi,

I did follow provided instructions and apt update keeps complaining about keys.
However the nvidia repository URLs are different from the ones listed in this thread.

Any idea how I could fix that?

Thank you,
David

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 208CE844D9F220AD
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 208CE844D9F220AD
W: Failed to fetch http://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64/dists/focal/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 208CE844D9F220AD
W: Failed to fetch http://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64/dists/focal-updates/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 208CE844D9F220AD
W: Some index files failed to download. They have been ignored, or old ones used instead.

Hi @david.audrain
That is not the “CUDA” repository, that repository is for DGX “BaseOS” systems. Need to enroll D9F220AD pubkey, I believe they have a keyring package named nvidia-repo-setup.

1 Like

@david.audrain
You’ll need the new dgx-repo and new nvidia-repo-keys packages to provide the new pubkey and to update your /etc/apt/sources.list.d/dgx.list. Please see the DGX release notes:

Also, I’m unclear if you’re running the DGX Base OS stack, which provides the CUDA GPG key via the cuda-compute-repo package. It sounds like you’ve manually changed the keys already?

1 Like

Hi, @kmittman @roarmstrong @jwitsoe I have updated the gpg key packages for cuda & it is working fine. However, when I am going to update using sudo apt-get update it is showing error in cudnn because of outdated gpg key. Can you please tell me how can I update the public key for CUDNN as well? The error is like below-

$sudo apt-get update
Get:1 file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27  InRelease
Ign:1 file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27  InRelease
Get:2 file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27  Release [564 B]
Get:2 file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27  Release [564 B]
Err:3 file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27  Release.gpg
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
Hit:4 https://download.docker.com/linux/ubuntu focal InRelease                 
Hit:5 http://ppa.launchpad.net/linrunner/tlp/ubuntu focal InRelease            
Get:6 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]      
Hit:7 http://bd.archive.ubuntu.com/ubuntu focal InRelease                      
Get:8 http://bd.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]     
Get:9 http://bd.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]   
Get:10 http://security.ubuntu.com/ubuntu xenial-security InRelease [99.8 kB]   
Hit:11 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Fetched 436 kB in 25s (17.2 kB/s)               
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27  Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1
W: Target Translations (en_US) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1
W: Target Translations (en) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1
W: Failed to fetch file:/var/cudnn-local-repo-ubuntu2004-8.4.0.27/Release.gpg  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1
W: Target Translations (en_US) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1
W: Target Translations (en) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1

@roarmstrong

The container images for 11.0+ should all be live at this point.

I think it’s better to announce NVIDIA blog(https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/). Because, user does not know the impact of this problem from current article.

1 Like

Hi @raz.issa00
The cuDNN local repository installed on your system: /var/cudnn-local-repo-ubuntu2004-8.4.0.27 contains packages and repo metadata signed with the deprecated 7fa2af80 pubkey.

I would suggest removing old NVIDIA local repositories in /var/ from your system. For example,
sudo apt-get remove --purge "cudnn-local-repo*"

NOTE: this does not remove installed packages such as libcudnn8 and libcudnn8-dev from the system.

2 Likes

The container images for 11.0+ should all be live at this point.

I think it’s better to announce NVIDIA blog(https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/ ). Because, user does not know the impact of this problem from current article.

I also think it would be better to add this announcement in chapter “Working with containers” on the blog that the upstream Docker image has been updated so that users can easily find it.
Until I saw this post, I thought that all Docker images released at this stage would still need the GPG key update patch in the future.

I run sudo apt update && sudo apt full-upgrade -y every day. Today it failed for an CUDA related reason that I think is related to this key rotation. Could anyone guide me on resolving this issue? I can’t update or install new packages now.

I followed the steps in the blog:

sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb

Every time I run sudo apt-key del 7fa2af80 I get this output:

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK

I’m concerned that I can’t update due to apt-key deprecation or that I haven’t deleted and updated they older key fast enough. I’m also concerned the problem with the MergeList is due to something at the file /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2204_x86%5f64_Packages and wonder if there is a way I can reset that file somehow.

$ sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease               
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease              
Hit:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease            
Hit:6 https://repo.nordvpn.com/deb/nordvpn/debian stable InRelease             
Hit:7 https://ppa.launchpadcontent.net/yubico/stable/ubuntu jammy InRelease    
Reading package lists... Error!                            
W: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2204_x86%5f64_Packages
E: The package lists or status file could not be parsed or opened.
1 Like

Hi @MicahParks
Thank you for bring this to my attention. The default .deb compression changed from XZ to Zstandard in Ubuntu 22.04, which is not recognized by the build of the dpkg executable currently in use for updating the repository metadata.

We had four postings yesterday, one of which was for NCCL; these .deb packages are compressed with Zstandard, other packages in the repository continue use XZ compression.

Working to resolve this issue on our end, though it may also require users to delete /var/lib/apt/lists/*cuda_repos* after the repository metadata has been re-generated.

2 Likes

Hi. It would be great if CuDNN could also be added to the 22.04 repos.
None of the libcudnn8 files seem to be available for 22.04 yet on ::
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/
Thanks.

1 Like