[Benchmark] nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, , , ,
1

Check out how nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 achieved 56.96 tokens/sec on text generation on NVIDIA DGX Spark with vLLM!

View full benchmark at nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 - Spark Arena Benchmark

4

Great :) Trying to mimicking your recipe: did you test perhaps with the @eugr docker image too ? and where did you get the mod ? mods:

  • mods/nemotron-omni

thanks !

5

Hey @trithemius! Happy to share. Two things:

  1. Container: I’m using stock vllm/vllm-openai:v0.20.0-aarch64-cu130-ubuntu2404, not @eugr’s custom vllm-node build. I did try @eugr’s image initially, but hit a Marlin NVFP4 kernel issue on SM121 (negative-scale truncation producing UNK tokens). The stock 0.20.0 multi-arch image with --moe-backend marlin works correctly out of the box on the GB10 for this model. Recipe lives at ~/.spark-arena/spark-vllm-docker/recipes/nemotron-3-nano-omni-nvfp4.yaml and references container: vllm/vllm-openai:v0.20.0-aarch64-cu130-ubuntu2404.

  2. mods/nemotron-omni/: That one’s not upstream in @eugr’s repo. It’s a tiny local mod I added because the stock vllm-openai images don’t ship librosa/soundfile/decord, which the Omni’s Parakeet speech encoder needs. Just create the dir and drop in a one-liner run.sh:

$ cat ~/.spark-arena/spark-vllm-docker/mods/nemotron-omni/run.sh
#!/bin/bash
set -e
pip install --quiet vllm[audio]
chmod +x ~/.spark-arena/spark-vllm-docker/mods/nemotron-omni/run.sh

That adds ~10 s to the cold boot to pip-install the audio extras inside the container; the pip cache is mounted, so subsequent boots are fast.

The other key bits that matter for the 56.96 tok/s on this NVFP4 build are the Marlin env vars (VLLM_NVFP4_GEMM_BACKEND=marlin, VLLM_USE_FLASHINFER_MOE_FP4=0, VLLM_MARLIN_USE_ATOMIC_ADD=1) plus --mamba-ssm-cache-dtype float32 and --video-pruning-rate 0.5, which are all already in the recipe in @eugr’s repo if you want a reference.

Let me know if you want me to push the mod upstream as a PR. Happy to.

6

You don’t have to use Marlin for NVFP4 anymore. Default Flashinfer backend works equally well with recent builds now.

Nemotron Omni works out of the box in our community build too - I guess I need to add a recipe, but you can launch it with this command:

./launch-cluster.sh --solo \
exec vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
--port 8888 \
--trust-remote-code  \
--gpu-memory-utilization 0.8  \
--limit-mm-per-prompt '{"video": 1, "image": 1, "audio": 1}'  \
--media-io-kwargs '{"video": {"fps": 2,  "num_frames": 256}}'  \
--allowed-local-media-path=/   \
--enable-prefix-caching  \
--max-num-batched-tokens 32768  \
--reasoning-parser nemotron_v3 \
--enable-auto-tool-choice  \
--tool-call-parser qwen3_coder

Sorry guys, I’ve got a bit behind with the updates to the recipes/etc, but needed to step back from the project for a bit to catch up with other stuff. But I will get back to it very soon, stay tuned!

7
model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 pp2048 6625.34 ± 93.11 314.26 ± 4.34 309.38 ± 4.34 314.26 ± 4.34
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 tg32 60.10 ± 0.51 62.06 ± 0.52
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_pp @ d4096 5343.06 ± 3532.11 4534.59 ± 5619.30 4529.71 ± 5619.30 4534.59 ± 5619.30
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_tg @ d4096 60.29 ± 1.06 62.26 ± 1.10
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 pp2048 @ d4096 4467.19 ± 1279.75 515.74 ± 183.03 510.86 ± 183.03 515.74 ± 183.03
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 tg32 @ d4096 59.79 ± 0.25 61.74 ± 0.26
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_pp @ d16384 9866.60 ± 268.22 1769.63 ± 43.84 1764.75 ± 43.84 1769.63 ± 43.84
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_tg @ d16384 59.90 ± 0.73 61.86 ± 0.76
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 pp2048 @ d16384 4022.12 ± 203.10 515.38 ± 26.11 510.50 ± 26.11 515.38 ± 26.11
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 tg32 @ d16384 61.90 ± 3.71 63.92 ± 3.82
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_pp @ d32078 8892.66 ± 104.00 3848.19 ± 53.72 3843.30 ± 53.72 3848.19 ± 53.72
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_tg @ d32078 62.60 ± 1.41 64.64 ± 1.46
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 pp2048 @ d32078 3220.01 ± 479.77 657.03 ± 108.25 652.15 ± 108.25 657.03 ± 108.25
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 tg32 @ d32078 62.06 ± 0.99 64.08 ± 1.02
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_pp @ d65535 7415.09 ± 5.00 9398.44 ± 10.95 9393.56 ± 10.95 9398.44 ± 10.95
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ctx_tg @ d65535 63.88 ± 1.46 65.97 ± 1.51
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 pp2048 @ d65535 1955.06 ± 60.48 1053.44 ± 33.16 1048.56 ± 33.16 1053.44 ± 33.16
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 tg32 @ d65535 67.68 ± 5.14 69.90 ± 5.33

llama-benchy (0.3.7)
date: 2026-05-01 12:06:00 | latency mode: api

9

This is the performance we were expecting. Congratulations to everyone working on it.