Failed to build docker based on cuda image

Accelerated Computing NGC GPU Cloud Container: CUDA
1

Local build of cuda based container image always failed with the following error:

E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (343435 != 345084). Mirror sync in progress? [IP: 152.195.19.142 443]
Hashes of expected file:
- Filesize:345084 [weak]
- SHA256:e9af61c4b2f44d714b157c49e40e92f98f50c98ef7d0fa45fc52ddd47947549a
- SHA1:79bf92660b1b0560c26d0d517c78778cad8a51ce [weak]
- MD5Sum:5a2b1f6d478629f661cc4149f9f2a602 [weak]
Release file created at: Tue, 06 Jul 2021 23:02:03 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

A quick repro for this issue, just run the following bash file:

#!/bin/bash
set -e
docker build -t my-docker $@ -<<EOF
FROM nvidia/cuda:11.4.0-devel-ubuntu20.04
RUN apt-get update
EOF