Hi.
We have noticed that the site, https://developer.download.nvidia.com/, has started to enforce a certain kind of rate limiter.
This is currently making it unable to install any Python packages inside NVIDIA container images.
In NVIDIA container images (example: nvcr.io/nvidia/pytorch:24.06-py3), the default pip configuration looks both at both official PyPI (pypi.org) and NVIDIA NGC PyPI (pypi.ngc.nvidia.com).
When installing a package, pip will query both PyPI registries, and the query to the latter is very likely to fail (because it is not a full mirror of official PyPI and is missing most of the packages), yielding a 404 response.
Since pip installs packages while querying multiple packages in brief succession, it is very likely to trigger many 404 requests in a very short period.
It seems that the site https://developer.download.nvidia.com/ is making incoming requests hang after a short burst of non-2xx HTTP requests.
Therefore, any innocent pip install requests will trigger several 404s and eventually a hanging connection, which pip cancels with a 100-second timeout and retries multiple times in a loop, delaying installation several minutes to even hours.
This is causing a serious problem in our use cases where pip install is ran automatically without supervision.