18.04 CUDA docker image is broken

Steps:

  1. docker run --rm -it nvidia/cudagl:10.1-devel-ubuntu18.04
  2. apt-get update

Result:
W: GPG error: Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository ‘Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease’ is not signed.

3 Likes

Seems related to these topics:

3 Likes

similar issue in docker; I followed the key commands posted on the forum but still having the same issue

I have the following build followed by some additional RUN and COPY commands but the build fails

FROM nvidia/cuda:11.0-cudnn8-runtime-ubuntu18.04

RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y git lsb-core

RUN apt-key del 7fa2af80
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb

However the build still fails with a similar error as tho I have not rotated the key:
Sending build context to Docker daemon 43.94MB

Step 1/19 : FROM nvidia/cuda:11.0-cudnn8-runtime-ubuntu18.04
 ---> 848be2582b0a
Step 2/19 : RUN apt-get update &&     apt-get upgrade -y &&     apt-get install -y git lsb-core
 ---> Running in b42917c47f5a
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease [1575 B]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Err:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [909 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2733 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1496 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [21.1 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:16 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [942 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2272 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3167 kB]
Get:21 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [73.8 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease' is not signed.

Can someone advise what I’m doing wrong?

**** UPDATE: mat.g was correct
They are rotating keys – see here for steps to resolve:

**** UPDATE2:
Welp ok turns out even adding the manual key fails so yes there’s definitely something wrong here.

RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub

root@033ada28c0a3:/# apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
Executing: /tmp/apt-key-gpghome.2G1j3RpS1J/gpg.1.sh --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
gpg: requesting key from 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub'
gpg: key A4B469963BF863CC: public key "cudatools <cudatools@nvidia.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@033ada28c0a3:/# apt-get update
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease [1575 B]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Hit:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Hit:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [709 kB]
Err:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
  File has unexpected size (952932 != 708701). Mirror sync in progress? [IP: ]
  Hashes of expected file:
   - Filesize:708701 [weak]
   - SHA256:3bb17f2151c764a6f868b36f175c82ba8deb79cd79160375ccdaacebf12a26c7
   - SHA1:c40cd8e703ef81fad35da4cec0fed58bf3b6c83a [weak]
   - MD5Sum:00af6704ab07a4620d61569ec0053961 [weak]
  Release file created at: Mon, 25 Apr 2022 23:02:48 +0000
Ign:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg
Reading package lists... Done
W: GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release: The following 
signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Same here,

I’m running the tensorflow/tensorflow:2.7.0-gpu docker file and added this at the beginning :

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-get install -y --no-install-recommends wget
RUN apt-key del 7fa2af80
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb
RUN apt-get update

Then I get this error on the update :

#12 3.762 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.199.20.126 443]
#12 3.762 Hashes of expected file:
#12 3.762 - Filesize:481481 [weak]
#12 3.762 - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
#12 3.762 - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
#12 3.762 - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
#12 3.762 Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
#12 3.762 E: Some index files failed to download. They have been ignored, or old ones used instead.

Here is the content of the sources.list cuda file, if this can somehow help :

#12 [ 9/15] RUN cat /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list
#12 sha256:2ffcda5a63c628ecf592fffe73cf2a6b382e4a7e1def1d0435adc54f47d84490
#12 0.443 deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] Index of /compute/cuda/repos/ubuntu2004/x86_64 /
#12 DONE 0.5s

UPDATE : The mirror sync seems finished, I do not have the error anymore
UPDATE 2 : I don’t know what I’m doing, maybe this doesn’t work after all
UPDATE 3 : I think that if the nvidia-ml.list isn’t removed it works anyway.

2 Likes

If you blow away the .lists, then I don’t think you need to install the key ring deb because you’re not even going to be using NVIDIA’s repositories anymore.

So you could just do…

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

…and that will “fix” the issue.

1 Like

When running RUN dpkg -i cuda-keyring_1.0-1_all.deb it adds a new list, having it and cuda.list created some conflicts. By removing it then adding the new one everything seems to be working.

I must say I’m not an expert, but it looks like it makes sense, no?

The recommended approach doesn’t work (it creates duplicated apt source). The alternate approach does. However, for a new cuda image, we also need to do the same for machine-learning repository. So this would work for a 18.04 cuda 10.2 image

Can’t copy paste the text because the forum doesn’t allow me to post reply with nvidia host (sigh…)

3 Likes

I’m not an expert on this either. Is it asking too much for NVIDIA to push updated images that work with their current repository keys?

It looks like a slightly modified version of francis.toupins dockerfile steps is working for me. The 18.04 image doesn’t have wget and apt isn’t aware of it without running apt-get update which doesn’t work unless you delete the lists.

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-get update && apt-get install -y --no-install-recommends wget
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb

Our Dockerfile doesn’t update any NVIDIA packages, and I don’t know if this is the “correct” solution.

I also can confirm that the above solution from dennis.pan which fetches the updated public keys seems to work.

RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

2 Likes

Yeah that looks legit. By removing the key and the lists before calling the update it seems to work. I’m still not sure if nvidia-ml.list needs to be removed though

another approach:

RUN apt-key del 7fa2af80 \
    && apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub \
    && apt-get update \
    && apt-get install -y libsm6 libgl1-mesa-glx libxext6 libfontconfig1 libxrender1 \
    && rm -rf /var/lib/apt/lists/*

Adjust the second to last line to reflect your Dockerfile package installations

@NVIDIA is there any plan to update the docker images or do all of your customers have to fix it in their own Dockerfiles?

Found that
nvidia/cuda:11.2.1-base-ubuntu20.04 is updated a few hours ago

I am facing the same issue, but with cuda:10.1-*-ubuntu18.04 images.
Will those be updated/fixed?

I just used the two public keys together, and this seems to resolve my issue

apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub && \
apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub && \
3 Likes

Hi, I just ignore the APT-KEY DEL and it worked.

FROM nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04

# RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub
2 Likes

Thanks. This one works for me.

The solution from @dennis.pan also worked for me.