Invalid public key for CUDA apt repository

Accelerated Computing NGC GPU Cloud Docker and NVIDIA Docker
#1

Using the cuda 11.2.2 docker container, apt update’s are no longer successful due to unavailable public keys for the cuda apt repository. See example below.

Attempting to load the key manually via: wget -qO- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - also does not resolve the issue.

It looks like this only started occurring today, is there a workaround or fix coming for this issue?

 docker run --rm -ti  docker.io/nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04 /bin/bash
root@787d050e4fa3:/# apt update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease [1575 B]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]    
Ign:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease   
Get:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Get:5 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease                          
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [73.8 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]                      
Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1496 kB]           
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]         
Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [909 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [21.1 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2733 kB]         
Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]                                 
Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:16 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]             
Get:17 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3167 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2272 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [942 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Reading package lists... Done                                
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
2 Likes
#2

same issue here

1 Like
#3

oem@JH:~$ sudo dpkg -i cuda-repo-ubuntu2004-11-6-local_11.6.2-510.47.03-1_amd64.deb
[sudo] password for oem:
Selecting previously unselected package cuda-repo-ubuntu2004-11-6-local.
(Reading database … 275893 files and directories currently installed.)
Preparing to unpack cuda-repo-ubuntu2004-11-6-local_11.6.2-510.47.03-1_amd64.deb …
Unpacking cuda-repo-ubuntu2004-11-6-local (11.6.2-510.47.03-1) …
Setting up cuda-repo-ubuntu2004-11-6-local (11.6.2-510.47.03-1) …
oem@JH:~$ sudo apt-key add /var/cuda-repo-ubuntu2004-11-6-local/7fa2af80.pub
OK
oem@JH:~$ sudo apt-get update
Get:1 file:/var/cuda-repo-ubuntu2004-11-1-local InRelease
Ign:1 file:/var/cuda-repo-ubuntu2004-11-1-local InRelease
Get:2 file:/var/cuda-repo-ubuntu2004-11-6-local InRelease
Ign:2 file:/var/cuda-repo-ubuntu2004-11-6-local InRelease
Get:3 file:/var/cuda-repo-ubuntu2004-11-1-local Release [564 B]
Get:4 file:/var/cuda-repo-ubuntu2004-11-6-local Release [564 B]
Get:3 file:/var/cuda-repo-ubuntu2004-11-1-local Release [564 B]
Get:4 file:/var/cuda-repo-ubuntu2004-11-6-local Release [564 B]
Get:5 file:/var/cuda-repo-ubuntu2004-11-6-local Release.gpg [836 B]
Get:5 file:/var/cuda-repo-ubuntu2004-11-6-local Release.gpg [836 B]
Get:6 http://dl.google.com/linux/chrome/deb stable InRelease [1,811 B]
Get:7 Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease [1,575 B]
Hit:8 Index of /repos/code/ stable InRelease
Hit:9 Index of /ubuntu focal InRelease
Get:11 Index of /ubuntu focal-updates InRelease [114 kB]
Hit:12 Index of /graphics-drivers/ppa/ubuntu focal InRelease
Get:13 file:/var/cuda-repo-ubuntu2004-11-6-local Packages [34.1 kB]
Get:14 Index of /ubuntu focal-security InRelease [114 kB]
Hit:15 Index of linux/ubuntu/ focal InRelease
Get:16 http://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,092 B]
Err:7 Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:17 Index of /ubuntu focal-backports InRelease [108 kB]
Get:18 Index of /ubuntu focal-updates/main amd64 Packages [1,750 kB]
Get:19 Index of /ubuntu focal-security/main i386 Packages [428 kB]
Get:20 Index of /ubuntu focal-updates/main i386 Packages [637 kB]
Get:21 Index of /ubuntu focal-updates/main amd64 DEP-11 Metadata [278 kB]
Get:22 Index of /ubuntu focal-updates/universe i386 Packages [678 kB]
Get:23 Index of /ubuntu focal-updates/universe amd64 Packages [920 kB]
Get:24 Index of /ubuntu focal-security/main amd64 Packages [1,422 kB]
Get:25 Index of /ubuntu focal-updates/universe amd64 DEP-11 Metadata [390 kB]
Get:26 Index of /ubuntu focal-updates/universe amd64 c-n-f Metadata [20.6 kB]
Get:27 Index of /ubuntu focal-updates/multiverse amd64 DEP-11 Metadata [944 B]
Get:28 Index of /ubuntu focal-backports/main Translation-en [10.2 kB]
Get:29 Index of /ubuntu focal-backports/main amd64 DEP-11 Metadata [7,984 B]
Get:30 Index of /ubuntu focal-backports/universe amd64 DEP-11 Metadata [30.8 kB]
Get:31 Index of /ubuntu focal-security/main Translation-en [246 kB]
Get:32 Index of /ubuntu focal-security/main amd64 DEP-11 Metadata [40.7 kB]
Get:33 Index of /ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB]
Get:34 Index of /ubuntu focal-security/universe amd64 DEP-11 Metadata [66.2 kB]
Get:35 Index of /ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB]
Get:36 Index of /ubuntu focal-security/multiverse amd64 DEP-11 Metadata [2,464 B]
Reading package lists… Done
W: GPG error: Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository ‘Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease’ is no longer signed.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
oem@JH:~$ gpg --recv-keys A4B469963BF863CC
gpg: keyserver receive failed: No data

1 Like
#4

Same issue

#5

Same error here, for a 11.3.1 cudnn docker image. Can’t do apt-get update.

Has anyone found a solution yet?

#6

Same problem here, just started today.

W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed.

1 Like
#7

Looks like nvidia did rotate the public keys and asking to update it accordingly. Not sure when the Docker images(For ex: nvidia/cuda:11.2.1-base-ubuntu18.04) maintained by nvidia will be updated
Some info here

#8

We’re in the process of updating the Docker images and Dockerfiles, but I don’t have an ETA at hand for any specific container image.

1 Like
#9

The following works in pytorch docker (cuda-keyring_1.0-1_all.deb file was downloaded with wget outside of docker first):

from pytorch/pytorch:1.10.0-cuda11.3-cudnn8-devel as builder

COPY ./cuda-keyring_1.0-1_all.deb cuda-keyring_1.0-1_all.deb

RUN rm /etc/apt/sources.list.d/cuda.list
&& rm /etc/apt/sources.list.d/nvidia-ml.list
&& dpkg -i cuda-keyring_1.0-1_all.deb

RUN apt-get update

#10

This worked for me, thank you! Key was indeed to download the key outside of docker.

#11

Same problem for me, just started today. Cannot do apt-get update :(

W: GPG error: Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

E: The repository ‘Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease’ is not signed.

#12

Adding the following seems to fix it:
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

5 Likes
#13

Unless RUN is an alias to sudo your solution would be better stated as;

$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

Though I suspect that second line is only if you have machine learning stuff.

18 Likes
#14

OK it worked

#15

It worked, thank you! I only ended up needing first command to get my stuff working again!

#16

Just wanted to pull your coat about the fact that the post in the Announcements category references a directory tree on the server for use with Ubuntu 22.04 “The Jammy Jellyfish”, however no such path is being served.

1 Like
#17

I am facing same issue as well. I started NVIDIA VM in azure and not able to run apt update because of publickey error. Also tried running sudo apt-key adv --recv-keys --keyserver keys.gnupg.net A4B469963BF863CC to get keys but its giving rror for No data… How do i fix it ?
I am using ubuntu 20.04

#18

Same issue here:

GPG error: Index of /compute/cuda/repos/ubuntu2004/x86_64 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

#19

Found myself in a Catch 22 using Docker container nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu20.04
This works:
RUN apt-key del 7fa2af80

But, you still can’t run apt update after that, because the container still holds the invalidated nvidia repos, so I had to:

RUN apt-key del 7fa2af80 \
    && rm /etc/apt/sources.list.d/nvidia-ml.list /etc/apt/sources.list.d/cuda.list

After that you can update and apt install wget, to do:

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb \
    && dpkg -i cuda-keyring_1.0-1_all.deb

All was fine for me after that! So I believe someone needs to update the containers themselves for this to start working again. Hope this helps someone.

2 Likes
#20

Hey. I have the same issue without root privileges. How to tackle this issue without sudo?

I am trying to build and store a singularity-container on sylabs. I am using clustersystem were I don’t have any privileges. Thanks

singularity build --remote library://A/B/C.sif:latest C.def

…

Reading package lists…
W: GPG error: Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository ‘REPOLINK InRelease’ is not signed.
FATAL: While performing build: while running engine: exit status 100