Model Download Timeout Due to Slow Network(parakeet-1-1b-rnnt-multilingual) — Possible Cache or Timeout Configuration Solutions?

AI & Data Science NVIDIA NIM Models
, , ,
1

Hello Team.

Our server has a slow internet connection, and after a certain period, it encounters the following error and terminates.
We are running this on a node equipped with 8× NVIDIA B200 GPUs, using the command below:

docker run -it --rm --name=parakeet-1-1b-rnnt-multilingual \
   --runtime=nvidia \
   --gpus '"device=0"' \
   --shm-size=8GB \
   -e NGC_API_KEY \
   -e NIM_HTTP_API_PORT=9000 \
   -e NIM_GRPC_API_PORT=50051 \
   -e NIM_TAGS_SELECTOR=mode=all \
   -e NIM_CACHE_PATH=/opt/nim/.cache \
   -v [local_path]:/opt/nim/.cache \
   -p 9000:9000 \
   -p 50051:50051 \
   nvcr.io/nim/nvidia/parakeet-1-1b-rnnt-multilingual:latest

Because of the slow network, the model download process fails after some time with the following error:

RuntimeError: Download Error: RequestError(reqwest::Error { kind: Decode, source: hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) })

As a result, the startup process stops with:

nimlib.exceptions.ManifestDownloadError: Error downloading manifest: RuntimeError: Download Error: RequestError(reqwest::Error { kind: Decode, source: hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) })

Our server times out due to the slow internet connection, and we’re looking for a way to prevent this issue or resume the download process instead of restarting everything.

Is there a way to use cached model files that were downloaded and stored on another server by moving them into the NIM_CACHE_PATH directory?
Alternatively, can I configure any environment variables to prevent the timeout issue caused by a slow network connection?
If there are any other potential solutions, I’d like to know about them as well.

Thank you in advance for your help and guidance.
Jade

2

Hi @hs.jang

A practical workaround is:

  1. On a machine with good internet, run the same NIM image once with a persistence cache (for example: -e NIM_CACHE_PATH=/opt/nim/.cache -v /fast-cache:/opt/nim/.cache) and let it fully start so the model is downloaded.
  2. Copy that cache directory (/fast-cache) to your slow server and mount it the same way there (-e NIM_CACHE_PATH=/opt/nim/.cache -v /slow-cache:/opt/nim/.cache)

With the model already in the cache, the container will start from local files and won’t need to download large artifacts over your slow link.

Hope this helps.