Updating the CUDA Linux GPG Repository Key

Technical Blogs & Events Technical Blog
#2

I have followed instructions posted in https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/ but still running into issue trying to install content form the repo sles15/x86_64 repo.

Digest verification failed when trying to install cuda-license-10-2.

Sample output:

602dd1eef4d7:/ # zypper in -y --repo cuda-sles15-x86_64 cuda-license-10-2
Loading repository data…
Reading installed packages…
Resolving package dependencies…

The following NEW package is going to be installed:
cuda-license-10-2

The following package has no support information from its vendor:
cuda-license-10-2

1 new package to install.
Overall download size: 22.0 KiB. Already cached: 0 B. After the operation, additional 59.6 KiB will be used.
Continue? [y/n/v/…? shows all options] (y): y
Retrieving package cuda-license-10-2-10.2.89-1.x86_64 (1/1), 22.0 KiB ( 59.6 KiB unpacked)
Retrieving: cuda-license-10-2-10.2.89-1.x86_64.rpm …[done]

Warning: Digest verification failed for file ‘cuda-license-10-2-10.2.89-1.x86_64.rpm’
[/var/tmp/AP_0xwFNjG5/cuda-license-10-2-10.2.89-1.x86_64.rpm]

expected c1e357bd47b05a401bd3fc9358ab00647937d9276ae14c11d275178003b8f6fe
but got 36b7b27765abac583e4244f6f13938b782516f47536f2f15f3916f024f8056fa

Accepting packages with wrong checksums can lead to a corrupted system and in extreme cases even to a system compromise.

However if you made certain that the file with checksum ‘36b7…’ is secure, correct
and should be used within this operation, enter the first 4 characters of the checksum
to unblock using this file on your own risk. Empty input will discard the file.

Unblock or discard? [36b7/…? shows all options] (discard): discard
Package cuda-license-10-2-10.2.89-1.x86_64 (cuda-sles15-x86_64) seems to be corrupted during transfer. Do you want to retry retrieval?
Abort, retry, ignore? [a/r/i] (a): a
Problem occurred during or after installation or removal of packages:
Installation has been aborted as directed.
Please see the above error message for a hint.

#3

@roarmstrong @kmittman The instructions from https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/ are not working for me. PLEASE HELP ASAP!!

I wanted to rebuild a Docker image, which is based on nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 by adding following instructions:

RUN apt-key del 7fa2af80 && \
	apt-get install wget && \
	cd /tmp && \
	wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \
	dpkg -i cuda-keyring_1.0-1_all.deb && \
	apt-get update

However apt-get update fails with:

#10 3.069 E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
#10 3.069 E: The list of sources could not be read.

Full log below:

#10 2.796 --2022-04-28 19:54:24--  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
#10 2.797 Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 152.199.20.126, 152.199.20.126
#10 2.867 Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|152.199.20.126|:443... connected.
#10 2.897 HTTP request sent, awaiting response... 200 OK
#10 2.907 Length: 4332 (4.2K) [application/x-deb]
#10 2.907 Saving to: 'cuda-keyring_1.0-1_all.deb'
#10 2.907
#10 2.907      0K ....                                                  100% 29.9M=0s
#10 2.907
#10 2.907 2022-04-28 19:54:24 (29.9 MB/s) - 'cuda-keyring_1.0-1_all.deb' saved [4332/4332]
#10 2.907
#10 2.935 Selecting previously unselected package cuda-keyring.
#10 2.941 (Reading database ... 13511 files and directories currently installed.)
#10 2.941 Preparing to unpack cuda-keyring_1.0-1_all.deb ...
#10 2.949 Unpacking cuda-keyring (1.0-1) ...
#10 2.982 Setting up cuda-keyring (1.0-1) ...
#10 3.069 E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
#10 3.069 E: The list of sources could not be read.
```
#4

Hi @user152836
We are still experiencing 48 stale metadata files on the CDN for some repositories including sles15/x86_64.

Hi @someuser
By default the add-apt-repository command appends repos to /etc/apt/sources.list rather than creating a new .list file in /etc/apt/sources.list.d/. Can you try something like this in your Dockerfile?

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04

RUN apt-get update && \
        sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list && \
        apt-key del 7fa2af80 && \
2 Likes
#5

China CDN is not updated correctly:
https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/

image
image1920×1800 237 KB

https://developer.download.nvidia.cn/compute/cuda/repos/debian11/x86_64/
image
image1920×1757 254 KB

We can see both old pkgs and cannot find the new key.
Downloading seems fine for the key, so could be an mismatch on the html page.

IMO big change like this should really be made transactionally.

Besides, the announcement suggest to remove old keys, but looks like other parts in the cuda ecosystem, e.g. cudnn/tensorrt/nvidia-docker/…, still rely on the old key, so as those pkgs in the old machine-learning repos.

#6

@roarmstrong @kmittman
I am facing same issue as well. I started NVIDIA VM in azure and not able to run apt update because of publickey error. Also tried running sudo apt-key adv --recv-keys --keyserver keys.gnupg.net A4B469963BF863CC to get keys but its giving rror for No data… How do i fix it ?
I am using ubuntu 20.04

i tried your method as well:
steps I followed:

~sudo apt-key del 7fa2af80
OK
~uname -a
Linux static-image-detector 5.8.0-1039-azure #42~20.04.1-Ubuntu SMP Thu Jul 15 14:11:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
~ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
2022-04-29 04:35:06 (350 MB/s) - ‘cuda-keyring_1.0-1_all.deb’ saved [4328/4328]
~ sudo dpkg -i cuda-keyring_1.0-1_all.deb
Setting up cuda-keyring (1.0-1) ...
~ sudo apt-get update
E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != 
E: The list of sources could not be read.

Even after following all the steps, it still gives error. Please explain how to fix the issue.

#7

Hi @Eyshika
Please delete the duplicate cuda .list file or entry in /etc/apt/sources.list

See my comment #11 in https://forums.developer.nvidia.com/t/the-repository-https-developer-download-nvidia-com-compute-cuda-repos-ubuntu1804-x86-64-release-is-not-signed/193764/11

Or the more comprehensive guide on GitHub: https://github.com/NVIDIA/cuda-repo-management/issues/4

1 Like
#8

Thanks @kmittman !
Actually I had to do the following instead (i.e. manipulate subfolder /etc/apt/sources.list.d instead)

sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list.d/*
sed -i '/developer\.download\.nvidia\.com\/compute\/machine-learning\/repos/d' /etc/apt/sources.list.d/*
1 Like
#9

Is there an ETA for when the nvidia base docker images (e.g. nvidia/cuda:10.2-devel-ubuntu18.04) will be updated with the new key? Is there currently any nvidia docker image online that works?

#10 
❯ sudo dpkg -L cuda-keyring
/.
/etc
/etc/apt
/etc/apt/preferences.d
/etc/apt/preferences.d/cuda-repository-pin-600
/etc/apt/sources.list.d
/etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list
/usr
/usr/share
/usr/share/doc
/usr/share/doc/cuda-keyring
/usr/share/doc/cuda-keyring/changelog.Debian.gz
/usr/share/keyrings
/usr/share/keyrings/cuda-archive-keyring.gpg

I installed cuda-keyring, but I found out that cuda-repository-pin-600
and cuda-ubuntu2004-x86_64.list were not installed. Is it the correct behavior?

#11

When doing dnf reposync on CentOS 7, only half of the packages (~2.4k/~4k) are updated.
All the rest ~1.6k are skipped.
We curl the pkg, and sha256 does not match what we had locally.
Wondering if the metadata on nvidia side wasn’t updated correctly to include all new pkgs.

#12

update: Seems to be working again as of this morning. Thanks NVidia :-)

#13

Hi @eldad.klaiman
The container images update is tracked on GitLab: Cuda Repo Signing Key Change in Progress (#158) · Issues · nvidia / container-images / cuda · GitLab

Hi @daisuke.nishimatsu
Can you provide any more information? I’m not able to reproduce

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb 
Selecting previously unselected package cuda-keyring.
(Reading database ... 4824 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.0-1_all.deb ...
Unpacking cuda-keyring (1.0-1) ...
Setting up cuda-keyring (1.0-1) ...

$ find / -name "*cuda*" 2>/dev/null | sort
/cuda-keyring_1.0-1_all.deb
/etc/apt/preferences.d/cuda-repository-pin-600
/etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list
/usr/share/doc/cuda-keyring
/usr/share/keyrings/cuda-archive-keyring.gpg
/var/lib/dpkg/info/cuda-keyring.conffiles
/var/lib/dpkg/info/cuda-keyring.list
/var/lib/dpkg/info/cuda-keyring.md5sums
/var/lib/dpkg/info/cuda-keyring.postinst
$ sudo apt-get update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1575 B]
Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [481 kB]
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease               
Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease                         
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Fetched 483 kB in 1s (674 kB/s)
Reading package lists... Done

Hi @xkszltl
Was this starting from a previous mirror snapshot? For RPM distros, all of the packages have been re-signed as part of the key rotation. I have validated that the packages match their metadata

$ git clone https://github.com/NVIDIA/cuda-repo-management.git
$ cd cuda-repo-management/
$ time ./repo-validate.sh --mirror=/path/to/compute/cuda/repos --distro=rhel7 --arch=x86_64 
==> Found 133 repo postings
==> Parsing repodata/c4d543a93df9f728da92503ac1d9e5a24e3327afb77ad4e42c06bbf0aa2999bf-primary.xml.gz
:: Found 4002 local files
:: Up-to-date
NsightSystems-linux-public-2019.3.7.5-3837e03.rpm [109266008] [84ef090c76]
NsightSystems-linux-public-2019.5.2.16-b54ef97.rpm [114658590] [0fc01ca9db]
cuda-10-0-10.0.130-1.x86_64.rpm [6272] [fa027476f1]
[...]
nvidia-driver-latest-dkms-510.47.03-1.el7.x86_64.rpm [23284004] [3482f3836f]
[...]
xorg-x11-drv-nvidia-libs-396.82-1.el7.x86_64.rpm [36076348] [83cf65d249]
yum-plugin-nvidia-0.3-1.el7.noarch.rpm [10124] [c1398b0f3a]
yum-plugin-nvidia-0.5-1.el7.noarch.rpm [10064] [040da661a7]

real	29m38.527s
#14

Following the instructions, I did the following:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb

and received the following response:

–2022-05-02 12:51:11-- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)… 152.195.19.142
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|152.195.19.142|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 4332 (4.2K) [application/x-deb]
Saving to: ‘cuda-keyring_1.0-1_all.deb’

cuda-keyring_1.0-1_ 100%[===================>] 4.23K --.-KB/s in 0s

2022-05-02 12:51:11 (92.5 MB/s) - ‘cuda-keyring_1.0-1_all.deb’ saved [4332/4332]

Then I :

rusty@apricot-dev:~$ sudo dpkg -i cuda-keyring_1.0-1_all.deb

and got:
Selecting previously unselected package cuda-keyring.
(Reading database … 302165 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.0-1_all.deb …
Unpacking cuda-keyring (1.0-1) …
Setting up cuda-keyring (1.0-1) …

But when I :

sudo apt update

I get errors :
Get:1 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local InRelease
Ign:1 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local InRelease
Get:2 file:/var/cuda-repo-ubuntu1804-10-2-local InRelease
Ign:2 file:/var/cuda-repo-ubuntu1804-10-2-local InRelease
Get:3 file:/var/visionworks-repo InRelease
Ign:3 file:/var/visionworks-repo InRelease
Get:4 file:/var/visionworks-sfm-repo InRelease
Ign:4 file:/var/visionworks-sfm-repo InRelease
Get:5 file:/var/visionworks-tracking-repo InRelease
Ign:5 file:/var/visionworks-tracking-repo InRelease
Get:6 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release [563 B]
Get:7 file:/var/cuda-repo-ubuntu1804-10-2-local Release [564 B]
Get:8 file:/var/visionworks-repo Release [1,999 B]
Get:9 file:/var/visionworks-sfm-repo Release [2,003 B]
Get:6 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release [563 B]
Get:10 file:/var/visionworks-tracking-repo Release [2,008 B]
Get:7 file:/var/cuda-repo-ubuntu1804-10-2-local Release [564 B]
Get:8 file:/var/visionworks-repo Release [1,999 B]
Get:9 file:/var/visionworks-sfm-repo Release [2,003 B]
Get:10 file:/var/visionworks-tracking-repo Release [2,008 B]
Err:11 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release.gpg
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
Err:12 file:/var/cuda-repo-ubuntu1804-10-2-local Release.gpg
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
Get:13 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64 InRelease [1,484 B]
Get:14 Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease [1,575 B]
Hit:17 Index of /ubuntu bionic InRelease
Hit:18 Index of linux/ubuntu/ bionic InRelease
Get:20 Index of /ubuntu bionic-updates InRelease [88.7 kB]
Get:21 Index of /compute/cuda/repos/ubuntu1804/x86_64 Packages [709 kB]
Get:22 Index of /ubuntu bionic-security InRelease [88.7 kB]
Get:23 Index of /ubuntu bionic-backports InRelease [74.6 kB]
Get:24 Index of /ubuntu bionic-updates/main i386 Packages [1,464 kB]
Get:25 Index of /ubuntu xenial-security InRelease [99.8 kB]
Get:26 Index of /ubuntu bionic-updates/main amd64 Packages [2,544 kB]
Get:27 Index of /ubuntu bionic-updates/main amd64 DEP-11 Metadata [297 kB]
Get:28 Index of /ubuntu bionic-updates/universe i386 Packages [1,606 kB]
Get:29 Index of /ubuntu bionic-updates/universe amd64 Packages [1,806 kB]
Get:30 Index of /ubuntu bionic-security/main amd64 Packages [2,201 kB]
Get:31 Index of /ubuntu bionic-updates/universe amd64 DEP-11 Metadata [301 kB]
Get:32 Index of /ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2,468 B]
Get:33 Index of /ubuntu bionic-backports/universe amd64 DEP-11 Metadata [9,272 B]
Get:34 Index of /ubuntu bionic-security/main i386 Packages [1,162 kB]
Get:35 Index of /ubuntu bionic-security/main amd64 DEP-11 Metadata [55.2 kB]
Get:36 Index of /ubuntu bionic-security/universe amd64 Packages [1,193 kB]
Get:37 Index of /ubuntu bionic-security/universe i386 Packages [1,016 kB]
Get:38 Index of /ubuntu bionic-security/universe amd64 DEP-11 Metadata [59.8 kB]
Get:39 Index of /ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2,464 B]
Get:40 Index of /ubuntu xenial-security/main amd64 DEP-11 Metadata [93.6 kB]
Fetched 14.9 MB in 3s (5,329 kB/s)
Reading package lists… Done
Building dependency tree
Reading state information… Done
24 packages can be upgraded. Run ‘apt list --upgradable’ to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/cuda-repo-ubuntu1804-10-2-local Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en_US) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Failed to fetch file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local/Release.gpg The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Failed to fetch file:/var/cuda-repo-ubuntu1804-10-2-local/Release.gpg The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en_US) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1

As you can imagine,

sudo apt upgrade

didn’t go well either:

sudo apt upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following package was automatically installed and is no longer required:
linux-hwe-5.4-headers-5.4.0-105
Use ‘sudo apt autoremove’ to remove it.
The following packages have been kept back:
cuda-drivers cuda-drivers-450 libnvidia-cfg1-470 libnvidia-compute-470 libnvidia-decode-470 libnvidia-encode-470 libnvidia-extra-470 libnvidia-fbc1-470 libnvidia-gl-470 libnvidia-ifr1-470
nvidia-compute-utils-470 nvidia-dkms-470 nvidia-driver-470 nvidia-kernel-common-470 nvidia-kernel-source-470 nvidia-utils-470 xserver-xorg-video-nvidia-470
The following packages will be upgraded:
libkeyutils1 libnvidia-common-470 libxnvctrl0 nsight-systems-2021.1.3 nvidia-modprobe nvidia-settings ubuntu-advantage-tools
7 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 249 MB of archives.
After this operation, 52.2 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 Index of /compute/cuda/repos/ubuntu1804/x86_64 libnvidia-common-470 470.103.01-0ubuntu1 [10.3 kB]
Get:2 Index of /compute/cuda/repos/ubuntu1804/x86_64 libxnvctrl0 510.47.03-0ubuntu1 [20.5 kB]
Get:3 Index of /compute/cuda/repos/ubuntu1804/x86_64 nsight-systems-2021.1.3 2021.1.3.14-b695ea9 [248 MB]
Get:4 Index of /ubuntu bionic-updates/main amd64 ubuntu-advantage-tools amd64 27.7~18.04.1 [712 kB]
Get:5 Index of /ubuntu bionic-updates/main amd64 libkeyutils1 amd64 1.5.9-9.2ubuntu2.1 [8,764 B]
Get:6 Index of /compute/cuda/repos/ubuntu1804/x86_64 nvidia-settings 510.47.03-0ubuntu1 [894 kB]
Get:7 Index of /compute/cuda/repos/ubuntu1804/x86_64 nvidia-modprobe 510.47.03-0ubuntu1 [19.8 kB]
Fetched 249 MB in 7s (36.2 MB/s)
Preconfiguring packages …
(Reading database … 302170 files and directories currently installed.)
Preparing to unpack …/0-ubuntu-advantage-tools_27.7~18.04.1_amd64.deb …
Unpacking ubuntu-advantage-tools (27.7~18.04.1) over (27.6~18.04.1) …
Preparing to unpack …/1-libkeyutils1_1.5.9-9.2ubuntu2.1_amd64.deb …
Unpacking libkeyutils1:amd64 (1.5.9-9.2ubuntu2.1) over (1.5.9-9.2ubuntu2) …
Preparing to unpack …/2-libnvidia-common-470_470.103.01-0ubuntu1_all.deb …
Unpacking libnvidia-common-470 (470.103.01-0ubuntu1) over (470.103.01-0ubuntu0.18.04.1) …
Preparing to unpack …/3-libxnvctrl0_510.47.03-0ubuntu1_amd64.deb …
Unpacking libxnvctrl0:amd64 (510.47.03-0ubuntu1) over (470.57.01-0ubuntu0.18.04.1) …
Preparing to unpack …/4-nsight-systems-2021.1.3_2021.1.3.14-b695ea9_amd64.deb …
update-alternatives: removing manually selected alternative - switching nsys to auto mode
update-alternatives: removing manually selected alternative - switching nsys-ui to auto mode
update-alternatives: using /opt/nvidia/nsight-systems/2021.2.3/host-linux-x64/nsys-ui to provide /usr/local/bin/nsys-ui (nsys-ui) in auto mode
Unpacking nsight-systems-2021.1.3 (2021.1.3.14-b695ea9) over (2021.1.3.12-5261246) …
Preparing to unpack …/5-nvidia-settings_510.47.03-0ubuntu1_amd64.deb …
Unpacking nvidia-settings (510.47.03-0ubuntu1) over (470.57.01-0ubuntu0.18.04.1) …
Preparing to unpack …/6-nvidia-modprobe_510.47.03-0ubuntu1_amd64.deb …
Unpacking nvidia-modprobe (510.47.03-0ubuntu1) over (450.115-0ubuntu1) …
Setting up ubuntu-advantage-tools (27.7~18.04.1) …
Installing new version of config file /etc/logrotate.d/ubuntu-advantage-tools …
Setting up nsight-systems-2021.1.3 (2021.1.3.14-b695ea9) …
update-alternatives: using /opt/nvidia/nsight-systems/2021.1.3/target-linux-x64/nsys to provide /usr/local/bin/nsys (nsys) in auto mode
update-alternatives: error: alternative path /opt/nvidia/nsight-systems/2021.1.3/host-linux-x64/nsight-sys doesn’t exist
update-alternatives: error: no alternatives for nsight-sys
update-alternatives: using /opt/nvidia/nsight-systems/2021.1.3/host-linux-x64/nsys-ui to provide /usr/local/bin/nsys-ui (nsys-ui) in manual mode
Setting up libnvidia-common-470 (470.103.01-0ubuntu1) …
Setting up nvidia-modprobe (510.47.03-0ubuntu1) …
Setting up libkeyutils1:amd64 (1.5.9-9.2ubuntu2.1) …
Setting up libxnvctrl0:amd64 (510.47.03-0ubuntu1) …
Setting up nvidia-settings (510.47.03-0ubuntu1) …
Processing triggers for man-db (2.8.3-2ubuntu0.1) …
Processing triggers for gnome-menus (3.13.3-11ubuntu1.1) …
Processing triggers for mime-support (3.60ubuntu1) …
Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) …
Processing triggers for libc-bin (2.27-3ubuntu1.5) …
rusty@apricot-dev:~$ sudo apt update
Get:1 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local InRelease
Ign:1 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local InRelease
Get:2 file:/var/cuda-repo-ubuntu1804-10-2-local InRelease
Ign:2 file:/var/cuda-repo-ubuntu1804-10-2-local InRelease
Get:3 file:/var/visionworks-repo InRelease
Ign:3 file:/var/visionworks-repo InRelease
Get:4 file:/var/visionworks-sfm-repo InRelease
Ign:4 file:/var/visionworks-sfm-repo InRelease
Get:5 file:/var/visionworks-tracking-repo InRelease
Ign:5 file:/var/visionworks-tracking-repo InRelease
Get:6 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release [563 B]
Get:7 file:/var/cuda-repo-ubuntu1804-10-2-local Release [564 B]
Get:8 file:/var/visionworks-repo Release [1,999 B]
Get:9 file:/var/visionworks-sfm-repo Release [2,003 B]
Get:6 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release [563 B]
Get:10 file:/var/visionworks-tracking-repo Release [2,008 B]
Get:7 file:/var/cuda-repo-ubuntu1804-10-2-local Release [564 B]
Get:8 file:/var/visionworks-repo Release [1,999 B]
Get:9 file:/var/visionworks-sfm-repo Release [2,003 B]
Get:10 file:/var/visionworks-tracking-repo Release [2,008 B]
Get:11 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64 InRelease [1,484 B]
Hit:12 Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease
Hit:13 Index of /ubuntu bionic InRelease
Hit:14 Index of linux/ubuntu/ bionic InRelease
Err:15 file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release.gpg
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
Hit:16 Index of /ubuntu bionic-updates InRelease
Hit:17 Index of /ubuntu bionic-backports InRelease
Err:18 file:/var/cuda-repo-ubuntu1804-10-2-local Release.gpg
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
Hit:19 Index of /ubuntu bionic-security InRelease
Hit:22 Index of /ubuntu xenial-security InRelease
Fetched 1,484 B in 1s (992 B/s)
Reading package lists… Done
Building dependency tree
Reading state information… Done
17 packages can be upgraded. Run ‘apt list --upgradable’ to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/cuda-repo-ubuntu1804-10-2-local Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en_US) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Failed to fetch file:/var/cuda-repo-cross-aarch64-ubuntu1804-10-2-local/Release.gpg The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Failed to fetch file:/var/cuda-repo-ubuntu1804-10-2-local/Release.gpg The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en_US) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1
W: Target Translations (en) is configured multiple times in /etc/apt/sources.list:54 and /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list:1

None of these issues appeared prior to the GPG key being changed.
How does this get fixed?

Thanks

#15

Hi @user152836
Can you try accessing the sles15/x86_64 again? It should be working on the CDN since this weekend.

Hi @xkszltl
Right, the files exist on the CDN but the index HTML page was not re-generated, I’ll try to get that fixed today.

Hi @rusty2
The local repo (cuda-repo-cross-aarch64-ubuntu1804-10-2-local) already installed on your system continues to use the 7fa2af80 key.

So I suggest removing the local cross repo from your system ( sudo apt-get remove --purge "cuda-repo-cross-aarch64-ubuntu1804-10-2-local*" ), otherwise you will need to re-enroll the old key.

#16

Thanks. Will deleting the local repo require further action to continue programming my TX2?

#17

Can nvidia update ALL old docker images with the right key? I am building tensorflow serving and it uses

FROM nvidia/cuda:11.2.1-base-ubuntu18.04 as base_build
...
RUN apt-get update && \
        apt-get install -y --no-install-recommends libnvinfer7=${TF_TENSORRT_VERSION}-1+cuda11.1 \
        libnvinfer-dev=${TF_TENSORRT_VERSION}-1+cuda11.1 \
        libnvinfer-plugin-dev=${TF_TENSORRT_VERSION}-1+cuda11.1 \
        libnvinfer-plugin7=${TF_TENSORRT_VERSION}-1+cuda11.1 \

and added

RUN rm /etc/apt/sources.list.d/cuda.list && \
    rm /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-key del 7fa2af80 && apt-get update && apt-get install -y --no-install-recommends wget && \
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \
    dpkg -i cuda-keyring_1.0-1_all.deb

It doesn’t work though as I am hitting

16:46:43  Reading state information...
16:46:43  e[91mE: Unable to locate package libnvinfer7
16:46:43  E: Version '7.2.2-1+cuda11.1' for 'libnvinfer-dev' was not found
16:46:43  E: Version '7.2.2-1+cuda11.1' for 'libnvinfer-plugin-dev' was not found
16:46:43  E: Unable to locate package libnvinfer-plugin7
16:46:43  E: Unable to locate package libnvonnxparsers7
16:46:43  E: Unable to locate package libnvparsers7

Any ideas?

#18

Hi @tianhaitong
TensorRT (libnvinfer) version 8.x and newer is available in the CUDA repository (3bf863cc.pub)
However TensorRT version 7.x is only available in the defunct Machine Learning repository (7fa2af80.pub)

Upgrading to a newer version is recommended. However, if TensorRT 7.x is still required then do not include:

  • rm /etc/apt/sources.list.d/nvidia-ml.list
  • apt-key del 7fa2af80
#19

Updating imagePullPolicy: Always for GPU operator doesn’t seem to have worked for me, I still get the error:

nvidia driver modules are not yet loaded, invoking runc directly
Creating directory NVIDIA-Linux-x86_64-470.82.01
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 470.82.01..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.
WARNING: You specified the '--no-kernel-module' command line option, nvidia-installer will not install a kernel module as part of this driver installation, and it will not remove existing NVIDIA kernel modules not part of an earlier NVIDIA driver installation.  Please ensure that an NVIDIA kernel module matching this driver version is installed separately.
========== NVIDIA Software Installer ==========
Starting installation of NVIDIA driver version 470.82.01 for Linux kernel version 5.4.0-109-generic
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...
Checking NVIDIA driver packages...
Updating the package cache...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is no longer signed.
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...

When inspecting the manifest, it looks like the clusterpolicy was applied correctly.

...
  containers:
  - args:
    - init
    command:
    - nvidia-driver
    image: nvcr.io/nvidia/driver:470.82.01-ubuntu20.04
    imagePullPolicy: Always
    name: nvidia-driver-ctr
    resources: {}
...

And when I describe the pod, it seems to have been pulling correctly.

  Normal   Pulling    4m5s (x4 over 6m48s)  kubelet            Pulling image "nvcr.io/nvidia/driver:470.82.01-ubuntu20.04"
  Normal   Created    4m3s (x4 over 6m46s)  kubelet            Created container nvidia-driver-ctr
  Normal   Started    4m3s (x4 over 6m45s)  kubelet            Started container nvidia-driver-ctr
  Normal   Pulled     4m3s                  kubelet            Successfully pulled image "nvcr.io/nvidia/driver:470.82.01-ubuntu20.04" in 1.595839537s
  Warning  BackOff    112s (x6 over 5m14s)  kubelet            Back-off restarting failed container

Edit for more info from pod description (actual container hash):

Containers:
  nvidia-driver-ctr:
    Container ID:  containerd://7746ab1052f1901ef392152f768cbb51c33dd9c6c13140a0499dba41005cded7
    Image:         nvcr.io/nvidia/driver:470.82.01-ubuntu20.04
    Image ID:      nvcr.io/nvidia/driver@sha256:2279b9303334642719a8e5ab2b614b587b7c6ca519d9ab4a869e2d2ed119a2ce
#20

Fix is to also update the image tag by updating the version defined in clusterpolicy to version: 470-signed

#21

@frenzi This does solve it for me as well. How did you find out?