DGX Spark / GB10 – nemotron-3-super-120b-a12b:1.8.0-variant fails downloading rl-030326-nvfp4 artifact (HTTP 400)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hello,

I am trying to deploy NVIDIA Nemotron-3-Super-120B-A12B on a DGX Spark (GPU detected as NVIDIA GB10) using the dedicated NIM container.

Environment

  • Hardware: NVIDIA DGX Spark / GB10

  • Container: nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:1.8.0-variant

  • Selected profile: 66f2cc1e52c372defe1bcf7eed8086a4c16022cefbddf2217f136f6bbcc47644

  • GPU detected by NIM: NVIDIA GB10

  • NGC CLI: 4.10.0 | 4.14.0

  • Docker login to nvcr.io: OK

  • NGC API key: regenerated as a Personal API Key with all required services enabled

  • NGC image visibility: OK

  • Docker pull: OK

What I validated

  1. The image is visible from NGC:
    ngc registry image info nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:1.8.0-variant

    Result:

    • Image found

    • arm64 digest present

    • metadata returned successfully

  2. Docker pull works:
    docker pull nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:1.8.0-variant

    Result:

    • Pull succeeds

    • Image is up to date

So the issue does not appear to be:

  • local Docker authentication

  • NGC API key validity

  • base image access

  • local machine configuration

  • GPU detection

Problem

NIM starts correctly, detects the GB10 GPU, selects the NVFP4 profile, and then fails when downloading the runtime artifact:

  • Artifact:
    nim/nvidia/nemotron-3-super-120b-a12b:rl-030326-nvfp4

  • URL failing:
    https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nemotron-3-super-120b-a12b/rl-030326-nvfp4/files

  • Error:
    HTTP 400 Bad Request

Conclusion so far

The dedicated NIM image is accessible, but the model artifact download fails after profile selection.
This suggests one of the following:

  • missing or broken entitlement for this specific artifact

  • artifact/package published incorrectly

  • backend NGC issue for this NVFP4 profile

Could you please confirm whether:

  1. rl-030326-nvfp4 is currently valid and accessible for DGX Spark / GB10 users

  2. there is any known entitlement requirement beyond standard NGC image access

  3. this is a known issue on the NVIDIA side

If useful, I can also provide the full startup logs.

Thanks.

Additional details

  • I am specifically targeting the NVFP4 deployment path.

  • I can successfully access the NIM image itself, but the failure happens only when NIM tries to fetch the internal model artifact after selecting the profile.

  • This looks reproducible.

2

Does it need an auth token or something?

3

Yes, I am already using a valid NGC Personal API Key.

I regenerated the key, enabled the required services, configured it with ngc config set, and also re-authenticated Docker with:

echo “$NGC_API_KEY” | docker login nvcr.io --username ‘$oauthtoken’ --password-stdin

I also confirmed that:

  • ngc registry image info nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:1.8.0-variant works
  • docker pull nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:1.8.0-variant works

So the base NGC authentication seems OK.
The failure only happens later, when NIM tries to download the internal runtime artifact rl-030326-nvfp4, which returns HTTP 400.

4

Hello neodarkcristal,

I attempted to reproduce your error with a fresh API key, but I was able to download the file just fine.

This could just be a transient error; would you mind trying one more time with a fresh API key and letting me know what happens?

5

I tried to run this using:

docker run -it --rm \
    --gpus all \
    --shm-size=16GB \
    -e NGC_API_KEY="XXXXXXXXXXXXXXXXXXX" \
    -e NIM_MODEL_PROFILE=66f2cc1e52c372defe1bcf7eed8086a4c16022cefbddf2217f136f6bbcc47644 \
    -v ~/ext/cache/nim:/opt/nim/.cache \
    -p 8000:8000 \
    nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:latest

It doesn’t have any issues downloading, but it crashes after loading the weights:

The full output is here:

https://gist.githubusercontent.com/DanTup/c1f06f53852a81d49f98f34e940f2654/raw/731b8d790cd9c9710b25a3e4c5973547327282fa/gistfile1.txt

I’m struggling to understand which part of the error is important, but possibly this bit is a failed assertion?

e[0;36m(EngineCore_DP0 pid=150)e[0;0m e[31mERRORe[0m e[90m03-12 21:36:59e[0m e[90m[core.py:946]e[0m     self.check(sympy.Le(left, right))

e[0;36m(EngineCore_DP0 pid=150)e[0;0m e[31mERRORe[0m e[90m03-12 21:36:59e[0m e[90m[core.py:946]e[0m   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/sizevars.py", line 427, in check

e[0;36m(EngineCore_DP0 pid=150)e[0;0m e[31mERRORe[0m e[90m03-12 21:36:59e[0m e[90m[core.py:946]e[0m     assert self.expect_true(expr)

e[0;36m(EngineCore_DP0 pid=150)e[0;0m e[31mERRORe[0m e[90m03-12 21:36:59e[0m e[90m[core.py:946]e[0m            ^^^^^^^^^^^^^^^^^^^^^^

e[0;36m(EngineCore_DP0 pid=150)e[0;0m e[31mERRORe[0m e[90m03-12 21:36:59e[0m e[90m[core.py:946]e[0m torch._inductor.exc.InductorError: AssertionError:

Any ideas?

Edit: Gemini told me to try:

    -e NIM_ENFORCE_EAGER=1 \
    -e NIM_RELAX_MEM_CONSTRAINTS=1 \
    -e VLLM_GPU_MEMORY_UTILIZATION=0.85 \

But this made no difference.

6

I got the same error, any solution would be welcome

7

So far the only solution that works with Nemotron 3 Super on DGX Spark is either our community Docker (GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub) or building TRT-LLM from source.

Both will give about the same performance and will not utilize hardware FP4 acceleration (as of now) as NVFP4 support on Spark is not fully implemented yet.

Another option is llama.cpp, but it is reported to be even slower, so vLLM is the way to go.

8

I’ll try this out next time, but it’s weird that nvidias own images for their own model are broken.. isn’t the whole point of the nim images supposed to be to simplify deploying a model with the right optimizations without having to do the work ourselves? 🤔

I actually tried that and it also failed (I posted the error in another thread somewhere) 🙃

9

Someone succeeded, but the speeds were no better than vLLM with Marlin, so no reason to go that route.

10

Yeah, I got it working via yourproject (and rediscovered that it runs interactively and not detached as I wanted 😄), but it was hallucinating really badly when I tested it quickly on open-webui.. I tried getting it to write some simple scripts so I could see the throughput, and it kept inventing packages that didn’t exist, even creating links to 404s, even after I called it out it kept doing it.

I don’t think it had any web access, so it couldn’t confirm them, but I’m surprised it continued to make things up (and claiming it had checked them) instead of just not trying to use them or telling me it doesn’t know of any.

(however, I don’t normally test things that way, so I can’t be sure the other models wouldn’t do the same.. I need to come up with a more consistent way to test models 🙃)

11

It can run detached if you specify -d flag

12

Oh maybe the issue was with logs? I think I had to add a pipe somewhere to get the logs to show up in docker logs. I’ll figure it out again next time I run this way. Thanks!

13

The logs are piping into docker logs now.

14

Ah, perfect - thanks! :)

15

Hello,

After 2 days and burning through tons of my GROK, ChatGPT 5.4, and Claude credits (making you guys money) I’ve narrowed this down and it does not appear to be a hardware or Docker issue on the ASUS GX10.

What I verified successfully:

  • NVIDIA driver loads correctly

  • GPU is visible on the host with nvidia-smi

  • Docker GPU access works with docker run --rm --gpus all ubuntu nvidia-smi

  • Running headless with GDM stopped does not change the core failure

The NIM container starts, but fails when requesting the model manifest from NGC:

https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nemotron-3-super-120b-a12b/rl-030326-fp8/files

The container error is:

ManifestDownloadError: Error downloading manifest: HTTP status client error (400 Bad Request)

Direct API tests show:

The NGC response includes:
API Key does not have correct permissions (or subscriptions)
and
Org context missing in uam request

I am logged in under my business account and using a valid unexpired API key created there.

Please confirm:

  1. Whether my account/org currently has the required entitlement for nim/nvidia/nemotron-3-super-120b-a12b

  2. Whether there is a known issue with org context / UAM mapping for this model

  3. Whether a different org-scoped key or entitlement is required for this NIM container

I’m frustrated!!!

-Jeremy Fissell

16

I’m seeing a very similar issue on an ASUS Ascent GX10 / GB10. I verified nvidia-smi on host, Docker GPU passthrough with docker run --rm --gpus all ubuntu nvidia-smi, and headless container launch. My failure occurs specifically when NIM requests the artifact manifest from NGC, where I get 403 Access Denied on the model endpoint and 400 Invalid Request with Org context missing / API Key does not have correct permissions (or subscriptions) on the files endpoint. So in my case it appears to be an NGC entitlement/org-context problem rather than only a local runtime issue.

17

Not all NIMs are free; some require an enterprise license. A few have Enterprise and standard/free options available which are named really similar.

Beyond that, not all NIMs with ARM64 images work on Spark.

18

Noted, I assumed that the NVidia software would prefer and run better on the NVidia NIN vs. Ollama… Hope I’m part of the club, I can where my NVidia Collared shirt if it helps… LOL :)

19

jwarner - I apricate your feedback! Thanks…

20

Hitting the same error… the trend here though seems like Spark is already abandoned