Unable to run nv-rerankqa-mistral-4b-v3 NIM Bad Gateway

AI & Data Science NVIDIA NIM Models
,
1

When I attempt to run the nv-rerankqa-mistral-4b-v3 NIM the first time, I get a Bad Gateway error. Is this an issue with my setup or is this on the NVIDIA side?

I get the following:

== NVIDIA Retriever Text Reranking NIM ==

NVIDIA Release 1.0.2 (build 94101e56e865f68d5dfadb5d02af31d64d86b8eb)
Model: nvidia/nv-rerankqa-mistral-4b-v3

Container image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This NIM container is governed by the NVIDIA AI Product Agreement here:
https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.
A copy of this license can be found under /opt/nim/LICENSE.

The use of this model is governed by the AI Foundation Models Community License
here: https://docs.nvidia.com/ai-foundation-models-community-license.pdf.

OTEL Logging handler requested, but Python logging auto-instrumentation not set up. Set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to enable logging auto-instrumentation.
downloading nim/nvidia/nv-rerankqa-mistral-4b-v3:3_tokenizer_v3
This could take a while.
–2025-03-03 21:36:15-- https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nv-rerankqa-mistral-4b-v3/versions/3_tokenizer_v3/zip
Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)… 35.166.186.15, 35.165.30.77
Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|35.166.186.15|:443… connected.
HTTP request sent, awaiting response… 502 Bad Gateway
2025-03-03 21:36:16 ERROR 502: Bad Gateway.

2025-03-03T21:36:16Z WARNING: tools.nim.ngc_models - Failed to download from nim/nvidia/nv-rerankqa-mistral-4b-v3:3_tokenizer_v3: Failure from NGC CLI -

2

Hi @eric.ferrara hmm, seems like the NIM isn’t running - can you do a docker ps to see if the container is running?

3

No, it is not running. This is the first time running it after pulling it, so it is trying to download and cache the necessary data, but failing. I am trying to follow the instructions outlined here: nv-rerankqa-mistral-4b-v3 Model by NVIDIA | NVIDIA NIM.

4

Can you verify you’re using a working NGC API key?

Can you also confirm you’re using this command to spin up the NIM?

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
docker run -it --rm \
    --gpus all \
    --shm-size=16GB \
    -e NGC_API_KEY \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -u $(id -u) \
    -p 8000:8000 \
    nvcr.io/nim/nvidia/nv-rerankqa-mistral-4b-v3:1.0.2```
5

Yes. That is exactly the command I am running.

6

Can you confirm if this issue has been resolved?

It seems like a temporary outage might have caused this.