docker run --rm -it nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04
root@f5752847b7b9:/# apt update
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Ign:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease
Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release [696 B]
Get:8 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:10 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B]
Get:11 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release.gpg [836 B]
Get:12 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B]
Get:13 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Ign:11 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release.gpg
Get:15 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [73.8 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [34.4 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2213 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [667 kB]
Get:19 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2400 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2837 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Get:23 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1434 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [26.7 kB]
Get:25 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [633 kB]
Reading package lists... Done
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <cudatools@nvidia.com>
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
Iām not able to reproduce, can you try again or provide some additional information, such as which server you are resolving?
$ docker run --rm -it nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04
Unable to find image 'nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04' locally
10.2-cudnn8-runtime-ubuntu18.04: Pulling from nvidia/cuda
e4ca327ec0e7: Pull complete
9ba32254bb3c: Pull complete
21bb6c18f647: Pull complete
9d1c8624fb43: Pull complete
612015fefa01: Pull complete
f674c21ac467: Pull complete
ef33eff4864d: Pull complete
45a5e679d304: Pull complete
Digest: sha256:6c6c7b207e0e167203df83f23761b6cdc90030d39e588fa94a44d75c52b20d44
Status: Downloaded newer image for nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04
root@2143aea3cc87:/# apt-get update
Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease
Ign:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease
Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release [696 B]
Get:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B]
Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release.gpg [836 B]
Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B]
Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages [808 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [73.8 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1437 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2400 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:19 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [633 kB]
Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [26.7 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2216 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [34.4 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2837 kB]
Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [667 kB]
Get:25 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Fetched 24.5 MB in 2min 23s (172 kB/s)
Reading package lists... Done
root@2143aea3cc87:/#
I have the problem and following is output from my apt update:
Err:20 Index of /compute/machine-learning/repos/ubuntu1804/x86_64 Release
404 Not Found [IP: 152.199.39.144 80]
Reading package lists⦠Done
E: The repository āIndex of /compute/machine-learning/repos/ubuntu1804/x86_64 Releaseā no longer has a Release file.
N: Updating from such a repository canāt be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
Hi @leeseng
There is currently a known CDN outage for the developer.download.nvidia.com domain with files showing as unavailable.
On a separate note, the NVIDIA Machine Learning repository is deprecated with cuDNN, NCCL, and TensorRT products now available in the CUDA repository.
For now you can remove the appropriate .list file in /etc/apt/sources.list.d/.
In the future, please report similar issues on GitHub: https://github.com/NVIDIA/cuda-repo-management/issues/1
Thanks a lot for the info! Iāll update my machine accordingly.
By the way, the TensorFlow guide still using the machine learning repo for nvinfer, as per ä½æēØ pip å®č£
TensorFlow
Hi @leeseng
The CDN outage has been resolved. Thanks, I have filed an internal bug to update the TensorFlow documentation
I got the same symptoms shown as below.
nvidia/cuda:11.2.0-cudnn8-devel-ubuntu18.04 [source]
$ docker run --rm -it nvidia/cuda:11.2.0-cudnn8-devel-ubuntu18.04 bash
root@2e3532b79ad3:/# apt update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease [1575 B]
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Ign:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease
Get:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B]
Get:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B]
Get:5 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [73.8 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [909 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1496 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2733 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [21.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2272 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3167 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [942 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Reading package lists... Done
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04
$ docker run --rm -it nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04
root@dd2a2bbf98df:/# apt update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease [1575 B]
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease
Get:5 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B]
Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B]
Get:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [73.8 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [909 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1496 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2733 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [21.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3167 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [942 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2272 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Reading package lists... Done
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
apt update worked fine yesterday, but today is not okay.
How can we solve the issue?
Hi @tj_tsai
It looks very similar but it is not the same symptom.
because the public key is not available: NO_PUBKEY A4B469963BF863CC
We rotated to a new GPG signing key today.
Please see instructions from one of these sources, to remove the old key and enroll the new key:
Hi @kmittman
After following the instructions on ubuntu1804/x86_64, we still failed to run apt update
root@7edc67760bfa:/# apt-key del 7fa2af80
OK
root@7edc67760bfa:/# wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
--2022-04-29 03:20:28-- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 152.199.39.144
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|152.199.39.144|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4332 (4.2K) [application/x-deb]
Saving to: 'cuda-keyring_1.0-1_all.deb'
cuda-keyring_1.0-1_all.deb 100%[=====================================================================================================================>] 4.23K --.-KB/s in 0s
2022-04-29 03:20:28 (94.6 MB/s) - 'cuda-keyring_1.0-1_all.deb' saved [4332/4332]
root@7edc67760bfa:/# dpkg -i cuda-keyring_1.0-1_all.deb
Selecting previously unselected package cuda-keyring.
(Reading database ... 12455 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.0-1_all.deb ...
Unpacking cuda-keyring (1.0-1) ...
Setting up cuda-keyring (1.0-1) ...
root@7edc67760bfa:/# apt update
E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.
apt-key list
Hi @tj_tsai
The apt-key command is deprecated and will be removed in Debian 12 and Ubuntu 22.10, see: DebianRepository/UseThirdParty - Debian Wiki
The cuda-keyring package contains 3 files:
- /usr/share/keyrings/cuda-archive-keyring.gpg
- /etc/apt/sources.list.d/cuda-
$distro-$arch.list - /etc/apt/preferences.d/cuda-repository-pin-600
This (1) installs the GPG key, (2) enables the repo, and (3) prioritizes the repository higher.
The āconflicting values setā indicates there are duplicate .list entries, either another cuda*.list file in /etc/apt/sources.list.d/ or appended to /etc/apt/sources.list.
If the latter, you can remove the duplicate entry with
sudo sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list
Itās the latter.
$ docker run --rm -it nvidia/cuda:11.2.0-cudnn8-devel-ubuntu18.04
root@667ff033b8a0:/# ls /etc/apt/sources.list.d/
cuda.list nvidia-ml.list
root@667ff033b8a0:/# cat /etc/apt/sources.list.d/cuda.list
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /
The image contains the file cuda.list by default.
$ dpkg -i cuda-keyring_1.0-1_all.deb
root@e300f860401f:/# dpkg -i cuda-keyring_1.0-1_all.deb
Selecting previously unselected package cuda-keyring.
(Reading database ... 12455 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.0-1_all.deb ...
Unpacking cuda-keyring (1.0-1) ...
Setting up cuda-keyring (1.0-1) ...
root@e300f860401f:/# apt update
E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.
- view cuda.list
$ cat /etc/apt/sources.list.d/cuda.list deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /
$ rm /etc/apt/sources.list.d/cuda.list
root@e300f860401f:/# rm /etc/apt/sources.list.d/cuda.list
root@e300f860401f:/# apt update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease [1575 B]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease
Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages [709 kB]
Hit:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release
Hit:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Err:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
Fetched 709 kB in 2s (397 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
25 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/Release.gpg The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Some index files failed to download. They have been ignored, or old ones used instead.
- view nvidia-ml.list
$ cat /etc/apt/sources.list.d/nvidia-ml.list deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /
$ rm /etc/apt/sources.list.d/nvidia-ml.list
root@e300f860401f:/# rm /etc/apt/sources.list.d/nvidia-ml.list
root@e300f860401f:/# apt update
Hit:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease
Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
25 packages can be upgraded. Run 'apt list --upgradable' to see them.
Done!
May I know if there is any plan to apply the changes/patches (including deleting the key 7fa2af80 and installing the new cuda-keyring package) back to original docker images (for example, nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04)?
Hi @tj_tsai
The container images update is tracked on GitLab: Cuda Repo Signing Key Change is causing package repo update failures (#158) Ā· Issues Ā· nvidia / container-images / cuda Ā· GitLab


