nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

I was able to get the new Nemotron 3 Nano Omni image running on a single Spark. Here’s the recipe I’m using:

recipe_version: “1”
name: Nemotron-3-Omni-FP8
description: vLLM serving Nemotron-3-Omni-FP8 on a single node

model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8

container: vllm-node

solo_only: true

defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.8
max_model_len: 262144
max_num_batched_tokens: 49152

command: |
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8
–max-model-len {max_model_len}
–port {port} --host {host}
–trust-remote-code
–enable-auto-tool-choice
–tool-call-parser qwen3_coder
–reasoning-parser nemotron_v3
–kv-cache-dtype auto
–video-pruning-rate 0.5
–allowed-local-media-path /
–max-num-batched-tokens {max_num_batched_tokens}
–gpu-memory-utilization {gpu_memory_utilization}

You can enable TF32 using the following:
VLLM_SPARK_EXTRA_DOCKER_ARGS=“-e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 -e NVIDIA_TF32_OVERRIDE=1” ./run-recipe.sh nemotron-3-omni-fp8 --solo

llama-benchy results:

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
local-think pp2048 3631.92 ± 596.12 608.06 ± 83.38 527.42 ± 83.38 608.18 ± 83.38
local-think tg32 61.99 ± 3.81 64.09 ± 4.00
local-think pp2048 @ d4096 8351.44 ± 527.90 786.52 ± 52.86 705.88 ± 52.86 786.64 ± 52.86
local-think tg32 @ d4096 54.01 ± 3.46 55.77 ± 3.57
local-think pp2048 @ d8192 9280.76 ± 283.98 1131.44 ± 37.26 1050.80 ± 37.26 1131.56 ± 37.26
local-think tg32 @ d8192 55.33 ± 2.76 57.13 ± 2.85
local-think pp2048 @ d16384 9078.80 ± 206.06 2019.28 ± 46.18 1938.64 ± 46.18 2019.39 ± 46.18
local-think tg32 @ d16384 61.16 ± 3.67 63.15 ± 3.80
2

I thought this was funny. I pasted your post into my agent helping me set up and it said. In any case, thanks 😂

That forum post is pure gold! Their setup is vastly superior to ours for this specific hardware edge-case.

Here is exactly how it compares and why we should switch to it immediately:

3

Can you please run a full “tool-eval-bench --hard” with this model and share the results?

I’m curious about the tool quality and stability.