Error with CUDA repository on Ubuntu 20.04

Since a few days, I suddenly get errors trying to use the CUDA repositories on Ubuntu 20.04.
Following the steps from the instructions on a fresh Ubuntu 20.04 system:

echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list
apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub

I get:

apt-get update
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release [697 B]
Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release.gpg [836 B]
Ign:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Ign:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Ign:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Err:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
  404  Not Found [IP: XXXXXX]
Fetched 330 kB in 1s (392 kB/s)
Reading package lists...
E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/by-hash/SHA256/751939d95516afc289908a19e447f0acc1506367f72ed356431a2b1a469cc8ca  404  Not Found [IP: XXXXXX]
E: Some index files failed to download. They have been ignored, or old ones used instead.

True enough, the file https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/by-hash/SHA256/751939d95516afc289908a19e447f0acc1506367f72ed356431a2b1a469cc8ca can not be fetched.

Checking https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Release , the file contains:

Acquire-By-Hash: yes

and the hash mentioned above. Am I the only one seeing this, or are the Ubuntu 20.04 repositories broken / is the Acquire-By-Hash feature on mistakenly even though the repo is deployed without it?

1 Like

Thanks for the hint. I have the same problem, and fixed it by adding [by-hash=no] to repository specification:

echo "deb [by-hash=no] http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list
3 Likes

Thanks, indeed that also works for me! It also fixes things on other Debian-based OSs, I observe the same on Debian 10.
I wonder where’s the best place to raise this as a bug to get it fixed in the actual repository?

I hit the same issue today. Open a support ticket 1 hour ago, seems to be fixed now.

$ sudo apt update
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 
Hit:3 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:6 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal InRelease
Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Hit:5 https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64 focal-updates InRelease
Hit:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release
Fetched 114 kB in 0s (258 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
459 packages can be upgraded. Run 'apt list --upgradable' to see them.

Thanks!
Indeed, it seems fixed for Focal now, it seems they have disabled the Acquire-By-Hash setting in the Release file. The Debian 10 repos, however, still have by-hash active:

$ curl -s https://developer.download.nvidia.com/compute/cuda/repos/debian10/x86_64/Release | tail -n1
Acquire-By-Hash: yes

So Debian 10 is still affected by this.

Hi @olifre, @grzegorz.gutowski, @bruno.travouillon
Yes, Acquire-By-Hash had been set to true for over 1 year for the CUDA Debian and Ubuntu repositories.

For some reason, a few weeks ago it stopped working, only in China, only if using .com.cn domain redirect: 404 in China when accessing cuda repo (#133) · Issues · nvidia / container-images / cuda · GitLab

Then a couple of days ago, it suddenly stopped working in other regions (with no CDN redirect). As such, for the cuDNN and TensorRT releases that went out yesterday (for select distros), I set this property to False.

Today, I fixed this for the remaining .deb repositories: Debian 10, Ubuntu 14.04, and WSL.

2 Likes

Many thanks! Indeed, that sounds quite mysterious, good to see it fixed!