Error apt-get updating from nvidia/cuda11.2.1-base-ubuntu20.04

[zax.rosenberg]

As of this morning, when I try to run apt-get update from the nvidia/cuda11.2.1-base-ubuntu20.04 base docker image, I’m seeing the following error:

 > [ 2/11] RUN apt-get update:                                                                                                                                               
...                                                                         
#5 0.873 Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64  InRelease                                                             
#5 0.958 Err:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release
#5 0.958   404  Not Found [IP: 152.195.19.142 443]
#5 1.046 Err:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64  Release
#5 1.046   404  Not Found [IP: 152.195.19.142 443]
...
#5 5.420 Reading package lists...
#5 6.583 E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release' does not have a Release file.
#5 6.583 E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64  Release' does not have a Release file.

This hasn’t happened before, and it looks like the docker image hasn’t changed in three months.

Anyone else having this issue? How can I resolve this?

[vibhu2]

I’m facing this issue as well.

[farfel314-nvidia]

All of the downloads regardless of OS seem to be down from the main site (CUDA Toolkit 11.6 Update 2 Downloads | NVIDIA Developer)

I can’t access any of the archived builds either.

[kimmjk]

Same here for the ubuntu 18.04. Is there a way to escalate this issue? Want to make sure Nvidia is aware of the problem

[patemotter]

Cannot install anything via pip either which is pulling from https://developer.download.nvidia.com/compute/redist/.

[cardman2014]

yes im seeing similar issues, many directories from the main package repos have gone missing.

[TomK]

Hello everyone,

We are experiencing an outage with our system, and are actively working to resolve the issue. Unfortunately we don’t have an ETA for when this will be fixed. I will update back here when this is working.

Thanks for your patience.

Tom K

[david185]

Tom, thanks for noticing! Looking forward to the fix as it’s blocking us. 🙏

[ram6]

Most likely you already have everything you need from nvidia and you can just remove their repo lists.
just add the following line to your Dockerfile before the apt-get update:

RUN rm /etc/apt/sources.list.d/cuda.list && rm /etc/apt/sources.list.d/nvidia-ml.list

this workaround works if you’re not apt-get installing something from nvidia

[TomK]

Hello,

I am happy to report that all assets have been reloaded and files are now available for download. Thanks for your patience.

Cheers,
Tom K