Updating the CUDA Linux GPG Repository Key

Hi @steven.ramboer and @xkszltl
I have filed an internal task to notify the cuDNN, NCCL, and TensorRT teams about this request for Ubuntu 22.04 .deb packages to be available (hopefully sooner than later). Also will ask about .deb packages for the Debian 11 repo too.

2 Likes

Fantastic. Looks like everything is back to normal. Thank you, @kmittman.

Hi all. For anyone annoyed by the apt-key warning :)

Easy fix …

Thanks! Had to keep stealing things from 20.04 to 11 and hope that’ll be fixed soon.

For whoever had the same challenge, here’s how you can do it properly, with priority to avoid overwriting debian 11’s own deb: Roaster/repo.sh at 460b2e126909448e0c285b3de4eaeb4ce5cdb009 · xkszltl/Roaster · GitHub

I’ve discovered this discussion while trying to update some CentOS 7 severs.

I have tried to follow the instructions here but they simply do not work. There is some circular logic happening and I have no idea how to break it:

I remove the bad key using sudo rpm -e gpg-pubkey-7fa2af80-576db785

I verify that it is gone with rpm -q gpg-pubkey --qf ‘%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n’

Yet when I try to install the latest nvidia driver, the bad key is RE-DOWNLOADED!

=============================================================================================================================================================================================================================================================
 Package                                                                           Arch                                               Version                                                         Repository                                        Size
=============================================================================================================================================================================================================================================================
Updating:
 nvidia-driver-latest-dkms                                                         x86_64                                             3:515.48.07-1.el7                                               cuda                                              23 M
Installing for dependencies:
 egl-wayland                                                                       x86_64                                             1.1.6-1.el7                                                     epel                                              29 k
Updating for dependencies:
 kmod-nvidia-latest-dkms                                                           x86_64                                             3:515.48.07-1.el7                                               cuda                                              30 M
 nvidia-driver-latest-dkms-NVML                                                    x86_64                                             3:515.48.07-1.el7                                               cuda                                             468 k
 nvidia-driver-latest-dkms-NvFBCOpenGL                                             x86_64                                             3:515.48.07-1.el7                                               cuda                                              59 k
 nvidia-driver-latest-dkms-cuda                                                    x86_64                                             3:515.48.07-1.el7                                               cuda                                             291 k
 nvidia-driver-latest-dkms-cuda-libs                                               x86_64                                             3:515.48.07-1.el7                                               cuda                                              54 M
 nvidia-driver-latest-dkms-devel                                                   x86_64                                             3:515.48.07-1.el7                                               cuda                                              19 k
 nvidia-driver-latest-dkms-libs                                                    x86_64                                             3:515.48.07-1.el7                                               cuda                                             177 M
 nvidia-modprobe-latest-dkms                                                       x86_64                                             3:515.48.07-1.el7                                               cuda                                              34 k
 nvidia-persistenced-latest-dkms                                                   x86_64                                             3:515.48.07-1.el7                                               cuda                                              36 k
 nvidia-xconfig-latest-dkms                                                        x86_64                                             3:515.48.07-1.el7                                               cuda                                              95 k

Transaction Summary
=============================================================================================================================================================================================================================================================
Install             (  1 Dependent package)
Upgrade  1 Package  (+10 Dependent packages)

Total size: 286 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda/packages/nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY                                                                                            
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Importing GPG key 0x7FA2AF80:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: ae09 fe4b bd22 3a84 b2cc fce3 f60f 4b3d 7fa2 af80
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Is this ok [y/N]: y


Public key for nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64.rpm is not installed


 Failing package is: 3:nvidia-driver-latest-dkms-cuda-515.48.07-1.el7.x86_64
 GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub

How can I get the correct key? RPM is supposed to download it according to the documentation yet it’s getting the old key. Please help!

Hi @lh2332
Yes, you will need to update the .repo file for RHEL-based, Fedora, and SUSE distros. The cuda-rhel7.repo file on your system is pointing at the old GPG key location, which is why it is re-downloading it.

This is mentioned very briefly in the CUDA Installation Guide

  1. Install the new CUDA public GPG key: The new GPG public key for the CUDA repository (RPM-based distros) is d42d0685.On a fresh installation of RHEL, the yum package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted.

For upgrades, you must also also fetch an updated .repo entry:

sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

It should like look this

$ cat /etc/yum.repos.d/cuda-rhel7.repo
[cuda-rhel7-x86_64]
name=cuda-rhel7-x86_64
baseurl=https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64
enabled=1
gpgcheck=1
gpgkey=https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/D42D0685.pub

Hi @kmittman

Thanks so much for the prompt reply.

I actually did that step, and the file is correct but the error persists.

-Lokke

Hi @kmittman

I got it to work by deleting the key, using wget to download the key, and then using rpmkeys to install the key manually.

I just wanted you to know the other method was failing

-Lokke

Do you happen to have other cuda repos listed under /etc/yum.repos.d?
Yours called cuda in log, but these days it’s called cuda-rhel7, so probably you have an old version as well and that’s the one complaining.

Ah yes there are other cuda repos. Since my fix worked, I’m not going to mess with that however but will keep it in mind.

Thanks for the help!

Go into /etc/apt/sources.list.d and delete cuda.list and cuda_learn.list (there should be another file there e.g. cuda-ubuntu2004-x86_64.list)

I’ve solved it with “gpgcheck” off :

vim /etc/yum.repos.d/cuda-rhel7.repo


[cuda-rhel7-x86_64]
name=cuda-rhel7-x86_64
baseurl=Index of /compute/cuda/repos/rhel7/x86_64
enabled=1
gpgcheck=0


#yum update

Hi @schwab1976
I do not recommend disabling the GPG signature check. This is an important security feature.

Updating the local gpg-pubkey on RHEL7-like distros can be achieved by refreshing the .repo file like so:

sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo

Below is a demonstration of the migration from the old GPG key to the new GPG key.

Please do not follow these steps.

$ podman run -it centos:7 /bin/bash
# yum-config-manager --disable updates >/dev/null
# curl https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo -o cuda-rhel7.repo
# sed -i 's|D42D0685\.pub|7fa2af80\.pub|' cuda-rhel7.repo
# yum-config-manager --add-repo cuda-rhel7.repo
# yum makecache

The deprecated GPG pubkey does not match the RPMs in the repository …

# yum install libnvjpeg-11-0
[...]
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda-rhel7-x86_64/packages/libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Public key for libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm is not installed
libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm                                                                                                                                                                                 | 2.0 MB  00:00:00     
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Importing GPG key 0x7FA2AF80:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: ae09 fe4b bd22 3a84 b2cc fce3 f60f 4b3d 7fa2 af80
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub
Is this ok [y/N]: y

Public key for libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm is not installed

 Failing package is: libnvjpeg-11-0-11.1.1.245-1.x86_64
 GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/7fa2af80.pub

Then force refresh the .repo file

# yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo                   
Loaded plugins: fastestmirror, ovl
adding repo from: https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo
grabbing file https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo to /etc/yum.repos.d/cuda-rhel7.repo
repo saved to /etc/yum.repos.d/cuda-rhel7.repo

Now packages can be installed

# yum install libnvjpeg-11-0
[...]
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/cuda-rhel7-x86_64/packages/libnvjpeg-11-0-11.1.1.245-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
Retrieving key from https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/D42D0685.pub
Importing GPG key 0xD42D0685:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: 610c 7b14 e068 a878 070d a4e9 9cd0 a493 d42d 0685
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/D42D0685.pub
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libnvjpeg-11-0-11.1.1.245-1.x86_64                                                                                                                                                                                         1/1 
  Verifying  : libnvjpeg-11-0-11.1.1.245-1.x86_64                                                                                                                                                                                         1/1 

Installed:
  libnvjpeg-11-0.x86_64 0:11.1.1.245-1                                                                                                                                                                                                        

Complete!

it seems the keys changed today. it is working!


wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin

sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub

sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"

sudo apt-get update

sudo apt-get -y install cuda

Your instructions to remove the old and install the new key lead to the same old key, at least for ubuntu2204/x86_64

Can you please correct your article under:

Chain of commands instructed there for Ubuntu:
$ sudo apt-key del 7fa2af80

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb

$ sudo dpkg -i cuda-keyring_1.0-1_all.deb
triggers then the result:
Selecting previously unselected package cuda-keyring.
(Reading database … 254715 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.0-1_all.deb …
Unpacking cuda-keyring (1.0-1) …
Setting up cuda-keyring (1.0-1) …

A deprecated public CUDA GPG key appear to be installed.
To remove the key, run this command:
sudo apt-key del 7fa2af80

Hi @philipp4
Please use cuda-keyring version 1.1
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
This has been updated in the CUDA Installation Guide https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#network-repo-installation-for-ubuntu

Has this procedure been updated? I have run into the same error message:
W GPG error: file:/var/cudnn-local-repo-ubuntu2204-8.9.1.23 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 26253C7BE7A7D88D
and I followed the procedure in the post, however the same GPG error is occurring.

The cuda-keyring package enables the “online” repository. It does not enroll a GPG public key for the “offline” local repos.

The local repo installers use ephemeral “one-time” keys, so please follow the copy instruction provided at install time, i.e. for CUDA 12.1

$ sudo dpkg -i cudnn-local-repo-ubuntu2204-8.9.1.23_1.0-1_amd64.deb 
Selecting previously unselected package cudnn-local-repo-ubuntu2204-8.9.1.23.
(Reading database ... 5258 files and directories currently installed.)
Preparing to unpack cudnn-local-repo-ubuntu2204-8.9.1.23_1.0-1_amd64.deb ...
Unpacking cudnn-local-repo-ubuntu2204-8.9.1.23 (1.0-1) ...
Setting up cudnn-local-repo-ubuntu2204-8.9.1.23 (1.0-1) ...

The public cudnn-local-repo-ubuntu2204-8.9.1.23 GPG key does not appear to be installed.
To install the key, run this command:
sudo cp /var/cudnn-local-repo-ubuntu2204-8.9.1.23/cudnn-local-E7A7D88D-keyring.gpg /usr/share/keyrings/

As such, simply adapt 1. Introduction — cuda-installation-guide-linux 12.2 documentation

sudo cp -v /var/cudnn-local-repo-ubuntu2204-8.9.1.23/cudnn-local-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update

Any chance at taking compliance seriously after 4 years? Especially with CMMC 2.0 here? Yes, FIPS matters.

1 Like

@BJSmithIEEE
A few possible workaround I’ve been using for similar things:

  • Use some custom scripts to query the 2nd/3rd level of dependency tree and kick out unwanted pkgs (e.g. nsight in this case).
  • Mirror their repo through https to ensure it’s authentic in transit, and repack/re-sign with compliant algo.