Hi all,
We wanted to know what Nemotron 3 Nano (30B total / 3.5B active, hybrid Mamba-2 + MoE) actually does on a single RTX 5090. So we ran it through NIM 2.0.12 with the NVFP4 profile and measured five things: memory, speed, long context, concurrency and answer quality. Nemotron Nano 9B v2, the other Nemotron option NIM offers for this card, is in the tables as a reference point.
TL;DR
- It fits with room to spare. The container takes about 21 GiB at 32 sequences and about 25 GiB at the image default of 256 (4k context). The FP8 build does not fit in 32 GB, so NVFP4 is the option on this card.
- ~300 tok/s for a single stream. That is 4.1× the 9B v2 in tokens per second. An answer comes back 2.9× faster, because Nemotron 3 Nano writes about 1.4× more tokens before it stops.
- The speed does not drop with context. 305 tok/s with 1k tokens of context, 308 tok/s with 120k.
- 128 concurrent chat requests inside the MLPerf Inference server latency target. With long RAG-shaped prompts the limit is 8.
- The answers are good, and the reasoning switch matters. With reasoning on: 95.7% on GSM8K and 88.0% on a 2,850-question MMLU sample. With reasoning off: 5.5 points lower on GSM8K and 12.3 lower on MMLU.
Concurrency numbers: synthetic prompts, closed loop. Chat = 200 tokens in / 200 out; RAG-shaped = 3,500 in / 500 out. MLPerf server target = p99 TTFT ≤ 2 s and p99 TPOT ≤ 100 ms.
One RTX 5090 · Nemotron 3 Nano via NIM 2.0.12 (NVFP4) · synthetic prompts, closed loop
Run it
docker run -d --gpus all -p 8000:8000 \
-e NGC_API_KEY \
-e NIM_MODEL_PROFILE=1fba9ecfcfb4cde28d4ce3fd55c40bca89a5a613e25e98f057befe6a7e99eada \
-e NIM_MAX_MODEL_LEN=8192 \
-v ~/.cache/nim:/opt/nim/.cache \
nvcr.io/nim/nvidia/nemotron-3-nano:2.0.12
That is the NVFP4 profile, everything else at the image default. To turn reasoning off for a request:
{"model": "nvidia/nemotron-3-nano",
"messages": [{"role": "user", "content": "..."}],
"chat_template_kwargs": {"enable_thinking": false}}
Results
| Nemotron 3 Nano (NVFP4, NIM 2.0.12) | Nemotron Nano 9B v2 (bf16, NIM 1.12.2) | |
|---|---|---|
| Memory the container adds at READY, at its smallest working budget (4k context, 32 sequences) | 21.0 GiB | 21.9 GiB |
| Single-stream generation, median | 305 tok/s | 73 tok/s |
Time to finish an answer, median (max_tokens 4096) |
7.31 s | 20.1 s |
| Generation at ~1k / ~14k / ~120k tokens of context | 305 / 315 / 308 tok/s | 72 / 45 / 52 tok/s |
| Most concurrent chat requests inside the MLPerf server target | 128 (image default, 256 sequences) | 8 (32 sequences) |
| Same, RAG-shaped prompts | 8 | 4 |
| GSM8K · MMLU sample, reasoning on | 95.7% · 88.0% | 95.2% · 84.0% |
Measured with AIPerf and lm-evaluation-harness 0.4.13 (temperature 0). Each concurrency sweep was run twice on different nights and reached the same limits, except the 9B v2 chat row, where we show the lower second run because its engine exited at 16 that night (see below). Per-run details, including one second run voided by a single failed request, are in the repo.
The two models differ in architecture, precision and NIM version. Read this as two deployment options on one card, not a race.
Concurrency as a curve
Chat-shaped requests, 200 in / 200 out · hollow: p99 time to first token > 2 s
The RAG-shaped sweep (3,500 in / 500 out) is the same picture with the ceiling much lower: Nemotron 3 Nano is inside the target up to 8 concurrent requests and hollow from 16, total throughput tops out just above 2,100 tok/s, and the 9B v2 is inside the target up to 4, hollow at 8 and 16, and exits at 32. That chart is in the first reply below (new-account limit: two images per post).
Each point is one concurrency level, labelled with that level. Higher means more tokens per second from the card in total; further left means each request’s own stream is slower. Solid line = the sweep the table takes its number from; faint points = the same sweep on another night; hollow marker = p99 time to first token over 2 s. Per-request speed is 1000 ÷ the mean inter-token latency in ms.
What NIM adds here
We ran Nemotron 3 Nano’s weights three ways: in the NIM container, in upstream vLLM 0.27.1 with only the model path, and in upstream vLLM with NIM’s own arguments. With NIM’s arguments, upstream vLLM was within 3% of NIM at every concurrency from 1 to 128; with the model path alone, within about 5%. GSM8K accuracy was the same.
That is what you should expect. For this model, NIM ships vLLM profiles only (support matrix). Upstream vLLM picks the same settings by itself, and NVIDIA’s Nemotron team has documented running Nemotron 3 Nano on vLLM (vLLM blog, December 2025). What NIM gives you is the validated profile and a pinned image. For what a hardware-specific profile adds on data-center GPUs, see NVIDIA’s NIM-off/NIM-on comparison on Nemotron 3 Ultra on the NVIDIA Technical Blog.
Things that will bite you
- The reasoning switch is different per generation. Nemotron 3 Nano:
chat_template_kwargs: {"enable_thinking": false}./no_thinkhas no effect on it. Nemotron Nano 9B v2: system message/no_think. Check that the switch worked by looking at answer length: on MMLU the median answer fell from 198 tokens to 7. - Memory is set by
max_num_seqs, not just context. A hybrid model keeps a fixed Mamba state per sequence (about 14.6 MiB each here). At the default of 256 sequences, that state sets the memory floor. To change the cap, useNIM_PASSTHROUGH_ARGS="--max-num-seqs 32";NIM_MAX_BATCH_SIZEdid not reach the engine. Confirm both in the startup log. - Warm up before you measure or demo. In some containers the first requests after READY ran at about half speed. Ten to twenty requests (about a minute) got it to a steady rate.
- On the 9B v2 image (NIM 1.12.2, the newest tag NGC offered), keep concurrency at 8 or below. Every container we drove to 16 or more concurrent requests eventually exited with the same Mamba-cache
IndexError. The same weights on upstream vLLM 0.30.0 did not exit at 16, 32 or 64.
A few honest notes
- One RTX 5090 on Docker Desktop / WSL2. The card is not on NIM’s verified-GPU list for this model.
- NVIDIA’s NIM 2.0.12 release notes list a known issue: the NVFP4 profiles of this model can return incorrect output on stop-string and math-conformance checks. Our GSM8K and MMLU results are as shown above, and upstream vLLM on the same weights matched NIM. But our task sets are not NVIDIA’s conformance checks. Where correct output is required, the release notes recommend BF16 or FP8, which need a bigger card.
- The prompts are synthetic. This is a latency-target measurement under synthetic load; it does not say how many people a deployed system can serve.
Everything is in the repo
Pre-registrations, raw results, analysis scripts, both concurrency runs and the full boundary of every run are in the Vol.2 README of NV-benchmark (tag vol2). Every number in this post has a row in the README’s claim-to-evidence table pointing to the file it comes from. This is the second post in the series; the first covers Llama 3.1 8B with NIM on the same card.
If you have Nemotron 3 Nano running on an RTX PRO 6000 or a DGX Spark, we’d love to compare numbers.
QuanTuring Inc. · member of the NVIDIA Inception Program
