Apt update failing on Ubuntu CUDA repo

DerekJuba · July 7, 2020, 6:21pm

When I run apt update on Ubuntu 18.04, I am getting

Ign:12 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
Get:12 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [185 kB]
Err:12 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
  File has unexpected size (213860 != 185496). Mirror sync in progress? [IP: 152.195.19.142 80]
  Hashes of expected file:
   - Filesize:185496 [weak]
   - SHA256:f7e866fdb738e4bda095feb338ba61fc8150f584fb4ab1b39a025f58c0e01932
   - SHA1:4462a72d5e4952c9cb747af9678d0a1d85857837 [weak]
   - MD5Sum:4be0b5ad7f6bbec51557955b4906e4eb [weak]
  Release file created at: Wed, 24 Jun 2020 19:48:37 +0000
Fetched 1,533 B in 2s (1,012 B/s)
Reading package lists... Done
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/Packages.gz  File has unexpected size (213860 != 185496). Mirror sync in progress? [IP: 152.195.19.142 80]
   Hashes of expected file:
    - Filesize:185496 [weak]
    - SHA256:f7e866fdb738e4bda095feb338ba61fc8150f584fb4ab1b39a025f58c0e01932
    - SHA1:4462a72d5e4952c9cb747af9678d0a1d85857837 [weak]
    - MD5Sum:4be0b5ad7f6bbec51557955b4906e4eb [weak]
   Release file created at: Wed, 24 Jun 2020 19:48:37 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

This is on a PC that used to update correctly. I also tried installing CUDA on a fresh Ubuntu 18.04 install and got the same result. Is the CUDA repo broken?

DerekJuba · July 7, 2020, 7:47pm

This seems to have been resolved. Perhaps the repos were being updated. (Maybe adding Ubuntu 20.04, support?)

jirka.borovec · October 4, 2020, 8:44am

the problem remains on multiple dockers…
see: PyTorch 1.7 Stable support by ydcjeff · Pull Request #3821 · Lightning-AI/lightning · GitHub

w01 · October 4, 2020, 4:18pm

Same/similar problem again, breaks automated builds. Looks like the package repository at https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/ is broken. Packages.gz was updated on 2020-09-23, but the hash files were not updated to match, causing apt-get update to fail.

xristos.d96 · October 5, 2020, 11:30am

@w01 did you find a way to fix this?

w01 · October 5, 2020, 5:17pm

From previous similar posts, the most common resolution seems to be to wait and hope someone from nvidia notices and fixes it, I don’t know if there is a way to report problems like this to them directly.

An easy way to reproduce this problem is using their container images, for any CUDA version:

docker run -it --rm nvidia/cudagl:11.0-devel-ubuntu18.04 bash
# Inside the container:
apt-get update

thickey · October 5, 2020, 6:23pm

same

adam.askew · October 5, 2020, 9:24pm

We are experiencing the same

bushn1989 · October 5, 2020, 10:06pm

Same here

w01 · October 19, 2020, 7:17pm

Checking today, 2020-10-19, this appears to have been fixed again. The example with a container image above now works.

…and apparently broken again a short time later:

docker run -it --rm nvidia/cudagl:11.0-devel-ubuntu18.04 bash
# Inside the container:
apt-get update

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1804_x86%5f64_Packages.lz4
E: The package lists or status file could not be parsed or opened.

florr.grosso · October 19, 2020, 8:40pm

Hi all, I’m experiencing the same issue. It was working correctly for me until an hour ago and then broke.

Any fix for this? It’s breaking my builds and I can’t solve it. I would really appreciate your help

manuel.ruidiaz · October 19, 2020, 8:53pm

Having the same issue as you.

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu2004_x86%5f64_Packages.lz4
E: The package lists or status file could not be parsed or opened.

manuel.ruidiaz · October 19, 2020, 9:35pm

Seems to be fixed now.

rcasero · October 19, 2020, 10:57pm

Seems broken to me (Ubuntu 20.04). This is the line in /etc/apt/sources.list.d/nvidia-machine-learning.list

deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /

Then, sudo apt update gives the error

Hit:23 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages
Err:23 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages
  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:194276 [weak]
   - SHA256:1bac2a7b1e94582f66ec7c05ea06f471913e35339ef2ebf5ba0381652809d6bf
   - SHA1:db6d26419ab88fa9eceddd7cbd5885e438f9c4b4 [weak]
   - MD5Sum:8d3d18923994daa10bc3e51babae223a [weak]
  Hashes of received file:
   - SHA256:0be17938292927f89ad8906636b1a431695247ecb2587328c10607d7d1d6d033
   - SHA1:ade1d6fa5ad1be0f9edcf816981b4243bd750c32 [weak]
   - MD5Sum:ab396558487574c5941b33fdae6315de [weak]
   - Filesize:188541 [weak]
  Release file created at: Wed, 23 Sep 2020 22:09:13 +0000
Fetched 7,995 B in 1s (5,502 B/s)
Reading package lists... Error!
E: Failed to fetch store:/var/lib/apt/lists/partial/developer.download.nvidia.com_compute_machine-learning_repos_ubuntu1804_x86%5f64_Packages  Hash Sum mismatch
   Hashes of expected file:
    - Filesize:194276 [weak]
    - SHA256:1bac2a7b1e94582f66ec7c05ea06f471913e35339ef2ebf5ba0381652809d6bf
    - SHA1:db6d26419ab88fa9eceddd7cbd5885e438f9c4b4 [weak]
    - MD5Sum:8d3d18923994daa10bc3e51babae223a [weak]
   Hashes of received file:
    - SHA256:0be17938292927f89ad8906636b1a431695247ecb2587328c10607d7d1d6d033
    - SHA1:ade1d6fa5ad1be0f9edcf816981b4243bd750c32 [weak]
    - MD5Sum:ab396558487574c5941b33fdae6315de [weak]
    - Filesize:188541 [weak]
   Release file created at: Wed, 23 Sep 2020 22:09:13 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1804_x86%5f64_Packages
E: The package lists or status file could not be parsed or opened.

chris_t0f · January 11, 2022, 6:58pm

Still Broken…

koshy.george.kg1 · July 13, 2022, 4:34am

E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/Packages.gz Hash Sum mismatch
Hashes of expected file:
- Filesize:815853 [weak]
- SHA256:66e891d82894f08ecedac13c4e058ce5ea21882032c66a5b4d981cc2552f94a1
- SHA1:7cc76dbdc9d32d50cf5a1d377c8c877bcef23dcd [weak]
- MD5Sum:dc953e306c3b53f23ccdbe433881a800 [weak]
Hashes of received file:
- SHA256:989834105adeb2987d44b3da1cf0bd621e4a86a260f5790dd2f865105d231efe
- SHA1:da0b54ecbb7f3dd76602bef508e9a8df901d1cd2 [weak]
- MD5Sum:aa04b5a6fb8ceb8c4c0676ce2c144903 [weak]
- Filesize:815853 [weak]
Last modification reported: Mon, 11 Jul 2022 23:01:57 +0000
Release file created at: Tue, 12 Jul 2022 15:01:44 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

koshy.george.kg1 · July 13, 2022, 4:40am

The above error was produced by

# dockerfile
FROM nvidia/cuda:11.6.1-cudnn8-devel-ubuntu18.0
RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-get clean && apt-get update

# build command
docker build --no-cache --tag my_tag .

jcook3701 · July 14, 2022, 7:37am

I am getting this same error on bare metal install of ubuntu 18. This started as a messed up gpg key. I followed the advice of the following article which left me with an error that signatures couldn’t be verified. This article solved that issue but I am now stuck at the same place as you. Nvidia please let us know how to proceed with fixing this hash error without removing hash checking for this repository. Thank you.

ariel.young · July 15, 2022, 8:04pm

As far as I can tell, this issue is still ongoing as of 13:00 PST on 15 July.