Updating the CUDA Linux GPG Repository Key

I run sudo apt update && sudo apt full-upgrade -y every day. Today it failed for an CUDA related reason that I think is related to this key rotation. Could anyone guide me on resolving this issue? I can’t update or install new packages now.

I followed the steps in the blog:

sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb

Every time I run sudo apt-key del 7fa2af80 I get this output:

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK

I’m concerned that I can’t update due to apt-key deprecation or that I haven’t deleted and updated they older key fast enough. I’m also concerned the problem with the MergeList is due to something at the file /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2204_x86%5f64_Packages and wonder if there is a way I can reset that file somehow.

$ sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease               
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease              
Hit:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease            
Hit:6 https://repo.nordvpn.com/deb/nordvpn/debian stable InRelease             
Hit:7 https://ppa.launchpadcontent.net/yubico/stable/ubuntu jammy InRelease    
Reading package lists... Error!                            
W: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2204_x86%5f64_Packages
E: The package lists or status file could not be parsed or opened.
1 Like

Hi @MicahParks
Thank you for bring this to my attention. The default .deb compression changed from XZ to Zstandard in Ubuntu 22.04, which is not recognized by the build of the dpkg executable currently in use for updating the repository metadata.

We had four postings yesterday, one of which was for NCCL; these .deb packages are compressed with Zstandard, other packages in the repository continue use XZ compression.

Working to resolve this issue on our end, though it may also require users to delete /var/lib/apt/lists/*cuda_repos* after the repository metadata has been re-generated.

2 Likes

Hi. It would be great if CuDNN could also be added to the 22.04 repos.
None of the libcudnn8 files seem to be available for 22.04 yet on ::
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/
Thanks.

1 Like

Would like to see cudnn/trt/nccl for debian 11 as well.

@kmittman Thank you for your response. Could you link the other postings so I can follow along for updates? My plan is to wait for a fix since I don’t want to change anything now and require more work when an official fix is available. In the meantime, I can’t update apt or install new packages, which is limiting during development, so I’ll want to be in the loop for any official fixes.

Hi @MicahParks
The Ubuntu 22.04 repository metadata issue has been resolved (the two NCCL .deb packages were removed for now).
Added a pinned topic here: Notice: Ubuntu 22.04 repository MergeList corruption

Unfortunately, this requires manually deleting malformed MergeList files on machines that ran apt-get update during the affected period.

$ sudo rm -v /var/lib/apt/lists/*cuda_repos*
$ sudo apt-get update

Sorry for the inconvenience.

2 Likes

Hi @steven.ramboer and @xkszltl
I have filed an internal task to notify the cuDNN, NCCL, and TensorRT teams about this request for Ubuntu 22.04 .deb packages to be available (hopefully sooner than later). Also will ask about .deb packages for the Debian 11 repo too.

2 Likes

Fantastic. Looks like everything is back to normal. Thank you, @kmittman.

Hi all. For anyone annoyed by the apt-key warning :)

Easy fix …

Thanks! Had to keep stealing things from 20.04 to 11 and hope that’ll be fixed soon.

For whoever had the same challenge, here’s how you can do it properly, with priority to avoid overwriting debian 11’s own deb: Roaster/repo.sh at 460b2e126909448e0c285b3de4eaeb4ce5cdb009 · xkszltl/Roaster · GitHub

I’ve discovered this discussion while trying to update some CentOS 7 severs.

I have tried to follow the instructions here but they simply do not work. There is some circular logic happening and I have no idea how to break it:

I remove the bad key using sudo rpm -e gpg-pubkey-7fa2af80-576db785

I verify that it is gone with rpm -q gpg-pubkey --qf ‘%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n’

Yet when I try to install the latest nvidia driver, the bad key is RE-DOWNLOADED!

=============================================================================================================================================================================================================================================================
 Package                                                                           Arch                                               Version                                                         Repository                                        Size
=============================================================================================================================================================================================================================================================
Updating:
 nvidia-driver-latest-dkms                                                         x86_64                                             3:515.48.07-1.el7                                               cuda                                              23 M
Installing for dependencies:
 egl-wayland                                                                       x86_64                                             1.1.6-1.el7                                                     epel                                              29 k
Updating for dependencies:
 kmod-nvidia-latest-dkms                                                           x86_64                                             3:515.48.07-1.el7                                               cuda                                              30 M
 nvidia-driver-latest-dkms-NVML                                                    x86_64                                             3:515.48.07-1.el7                                               cuda                                             468 k
 nvidia-driver-latest-dkms-NvFBCOpenGL                                             x86_64                                             3:515.48.07-1.el7                                               cuda                                              59 k
 nvidia-driver-latest-dkms-cuda                                                    x86_64                                             3:515.48.07-1.el7                                               cuda                                             291 k
 nvidia-driver-latest-dkms-cuda-libs                                               x86_64                                             3:515.48.07-1.el7                                               cuda                                              54 M
 nvidia-driver-latest-dkms-devel                                                   x86_64                                             3:515.48.07-1.el7                                               cuda                                              19 k
 nvidia-driver-latest-dkms-libs                                                    x86_64                                             3:515.48.07-1.el7                                               cuda                                             177 M
 nvidia-modprobe-latest-dkms                                                       x86_64                                             3:515.48.07-1.el7                                               cuda                                              34 k
 nvidia-persistenced-latest-dkms                                                   x86_64                                             3:515.48.07-1.el7                                               cuda                                              36 k
 nvidia-xconfig-latest-dkms                                                        x86_64                                             3:515.48.07-1.el7                                               cuda                                              95 k

Transaction Summary
=============================================================================================================================================================================================================================================================
Install             (  1 Dependent package)
Upgrade  1 Package  (+10 Dependent packages)

Total size: 286 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda/packages/nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY                                                                                            
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Importing GPG key 0x7FA2AF80:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: ae09 fe4b bd22 3a84 b2cc fce3 f60f 4b3d 7fa2 af80
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Is this ok [y/N]: y


Public key for nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64.rpm is not installed


 Failing package is: 3:nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64
 GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub

How can I get the correct key? RPM is supposed to download it according to the documentation yet it’s getting the old key. Please help!

Hi @lh2332
Yes, you will need to update the .repo file for RHEL-based, Fedora, and SUSE distros. The cuda-rhel7.repo file on your system is pointing at the old GPG key location, which is why it is re-downloading it.

This is mentioned very briefly in the CUDA Installation Guide

  1. Install the new CUDA public GPG key: The new GPG public key for the CUDA repository (RPM-based distros) is d42d0685.On a fresh installation of RHEL, the yum package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted.

For upgrades, you must also also fetch an updated .repo entry:

sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

It should like look this

$ cat /etc/yum.repos.d/cuda-rhel7.repo
[cuda-rhel7-x86_64]
name=cuda-rhel7-x86_64
baseurl=https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64
enabled=1
gpgcheck=1
gpgkey=https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/D42D0685.pub

Hi @kmittman

Thanks so much for the prompt reply.

I actually did that step, and the file is correct but the error persists.

-Lokke

Hi @kmittman

I got it to work by deleting the key, using wget to download the key, and then using rpmkeys to install the key manually.

I just wanted you to know the other method was failing

-Lokke

Do you happen to have other cuda repos listed under /etc/yum.repos.d?
Yours called cuda in log, but these days it’s called cuda-rhel7, so probably you have an old version as well and that’s the one complaining.

Ah yes there are other cuda repos. Since my fix worked, I’m not going to mess with that however but will keep it in mind.

Thanks for the help!

Go into /etc/apt/sources.list.d and delete cuda.list and cuda_learn.list (there should be another file there e.g. cuda-ubuntu2004-x86_64.list)

I’ve solved it with “gpgcheck” off :

vim /etc/yum.repos.d/cuda-rhel7.repo


[cuda-rhel7-x86_64]
name=cuda-rhel7-x86_64
baseurl=Index of /compute/cuda/repos/rhel7/x86_64
enabled=1
gpgcheck=0


#yum update

Hi @schwab1976
I do not recommend disabling the GPG signature check. This is an important security feature.

Updating the local gpg-pubkey on RHEL7-like distros can be achieved by refreshing the .repo file like so:

sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo

Below is a demonstration of the migration from the old GPG key to the new GPG key.

Please do not follow these steps.

$ podman run -it centos:7 /bin/bash
# yum-config-manager --disable updates >/dev/null
# curl https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo -o cuda-rhel7.repo
# sed -i 's|D42D0685\.pub|7fa2af80\.pub|' cuda-rhel7.repo
# yum-config-manager --add-repo cuda-rhel7.repo
# yum makecache

The deprecated GPG pubkey does not match the RPMs in the repository …

# yum install libnvjpeg-11-0
[...]
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda-rhel7-x86_64/packages/libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Public key for libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm is not installed
libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm                                                                                                                                                                                 | 2.0 MB  00:00:00     
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Importing GPG key 0x7FA2AF80:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: ae09 fe4b bd22 3a84 b2cc fce3 f60f 4b3d 7fa2 af80
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Is this ok [y/N]: y

Public key for libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm is not installed

 Failing package is: libnvjpeg-11-0-11.1.1.245-1.x86_64
 GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub

Then force refresh the .repo file

# yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo                   
Loaded plugins: fastestmirror, ovl
adding repo from: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo
grabbing file https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo to /etc/yum.repos.d/cuda-rhel7.repo
repo saved to /etc/yum.repos.d/cuda-rhel7.repo

Now packages can be installed

# yum install libnvjpeg-11-0
[...]
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda-rhel7-x86_64/packages/libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/D42D0685.pub
Importing GPG key 0xD42D0685:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: 610c 7b14 e068 a878 070d a4e9 9cd0 a493 d42d 0685
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/D42D0685.pub
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libnvjpeg-11-0-11.1.1.245-1.x86_64                                                                                                                                                                                         1/1 
  Verifying  : libnvjpeg-11-0-11.1.1.245-1.x86_64                                                                                                                                                                                         1/1 

Installed:
  libnvjpeg-11-0.x86_64 0:11.1.1.245-1                                                                                                                                                                                                        

Complete!

it seems the keys changed today. it is working!


wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin

sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub

sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"

sudo apt-get update

sudo apt-get -y install cuda