The title says it all. This looks like a match made in heaven for us.
Iโm dying to get home and test it.
Anyone have any insight/tips and tricks?
I am getting 19 tok/s using the recipe below. I am sure it could be optimized.
recipe_version: โ1โ
name: Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
description: |
vLLM serving NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 on a single DGX Spark node (TP=1).
Optimized with MTP speculative decoding (3 tokens) + Marlin NVFP4 backend for SM121 stability & speed.
Expected single-stream decode: 25-35+ tok/s on DGX Spark.
model: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
container: vllm-node
solo_only: true
mods:
env:
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: โ0โ
VLLM_USE_FLASHINFER_MOE_FP4: โ0โ
VLLM_NVFP4_GEMM_BACKEND: โmarlinโ
VLLM_TEST_FORCE_FP8_MARLIN: โ1โ
defaults:
port: 8000
host: โ0.0.0.0โ
tensor_parallel: 1
gpu_memory_utilization: 0.85
max_model_len: 131072
max_num_batched_tokens: 16384
max_num_seqs: 12
command: |
vllm serve nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
โtensor-parallel-size {tensor_parallel}
โhost {host}
โport {port}
โtrust-remote-code
โmoe-backend marlin
โmamba-backend flashinfer
โmamba_ssm_cache_dtype float16
โenable-mamba-cache-stochastic-rounding
โmamba-cache-philox-rounds 5
โasync-scheduling
โkv-cache-dtype fp8
โgpu-memory-utilization {gpu_memory_utilization}
โmax-model-len {max_model_len}
โmax-num-batched-tokens {max_num_batched_tokens}
โmax-num-seqs {max_num_seqs}
โenable-chunked-prefill
โenable-prefix-caching
โenable-expert-parallel
โtool-call-parser qwen3_coder
โreasoning-parser nemotron_v3
โenable-auto-tool-choice
โspeculative-config โ{โmethodโ:โmtpโ,โnum_speculative_tokensโ:3}โ
โgeneration-config vllm
Hello, after several attempts, all I obtain is that :
tool-eval-bench --spec-bench --base-url "http://127.0.0.1:8002" --api-key "sk-vllm-internal-acantic" --context-pressure 0.8 --depth "0 16384 32768"
๐ง Tool-Call Benchmark
Server: http://127.0.0.1:8002
Querying http://127.0.0.1:8002/v1/models โฆ โ nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 (alias: Acantic)
โ Warm-up complete (291 ms**)**
๐ Engine: vLLM 0.23.1rc1.dev899+g378a4be5d.d20260707
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ๐ฎ Speculative Decoding Benchmark โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 โ
โ tg=128 depth=[0, 16384, 32768] prompts=['filler', 'code', 'structured'] method=auto โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Prometheus /metrics acceptance-rate counters are server-wide aggregates. If other models are serving concurrent traffic on this endpoint, per-request acceptance rate measurements will be inaccurate. For clean measurements: use a single-model server with no concurrent load.
โ filler @ d0 29.2 eff t/s 29.0 stream t/s ฮฑ=78.1% waste=22% ฯ=2.3 win=3
โ code @ d0 31.5 eff t/s 31.3 stream t/s ฮฑ=60.1% waste=40% ฯ=1.8 win=3
โ structured @ d0 31.9 eff t/s 31.7 stream t/s ฮฑ=65.9% waste=34% ฯ=2.0 win=3
โ filler @ d16384 10.3 eff t/s 10.2 stream t/s ฮฑ=78.1% waste=22% ฯ=2.3 win=3
โ code @ d16384 30.3 eff t/s 30.1 stream t/s ฮฑ=57.4% waste=43% ฯ=1.7 win=3
โ structured @ d16384 34.9 eff t/s 34.6 stream t/s ฮฑ=75.2% waste=25% ฯ=2.3 win=3
โ filler @ d32768 8.9 eff t/s 8.8 stream t/s ฮฑ=69.9% waste=30% ฯ=2.1 win=3
โ code @ d32768 31.7 eff t/s 31.4 stream t/s ฮฑ=60.7% waste=39% ฯ=1.8 win=3
โ structured @ d32768 33.9 eff t/s 33.7 stream t/s ฮฑ=70.7% waste=29% ฯ=2.1 win=3
***Speculative Decoding Results***
โโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโโโโณโโโโโโโโโณโโโโโโโโณโโโโโโโโณโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโ
โ Prompt โ Depth โ Eff t/s โ ฮฑ % โ Waste โ ฯ len โ Win โ Draft t/s โ TTFT ms โ Total ms โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ filler โ 0 โ 29.2 โ 78.1% โ 22% โ 2.3 โ 3 โ 26.0 โ 6 โ 4,390 โ
โ code โ 0 โ 31.5 โ 60.1% โ 40% โ 1.8 โ 3 โ 34.0 โ 3 โ 4,062 โ
โ structured โ 0 โ 31.9 โ 65.9% โ 34% โ 2.0 โ 3 โ 32.2 โ 4 โ 4,011 โ
โ filler โ 16K โ 10.3 โ 78.1% โ 22% โ 2.3 โ 3 โ 9.2 โ 19 โ 12,447 โ
โ code โ 16K โ 30.3 โ 57.4% โ 43% โ 1.7 โ 3 โ 33.4 โ 3 โ 4,226 โ
โ structured โ 16K โ 34.9 โ 75.2% โ 25% โ 2.3 โ 3 โ 31.9 โ 3 โ 3,672 โ
โ filler โ 32K โ 8.9 โ 69.9% โ 30% โ 2.1 โ 3 โ 8.5 โ 59 โ 14,500 โ
โ code โ 32K โ 31.7 โ 60.7% โ 39% โ 1.8 โ 3 โ 33.4 โ 4 โ 4,048 โ
โ structured โ 32K โ 33.9 โ 70.7% โ 29% โ 2.1 โ 3 โ 32.6 โ 4 โ 3,774 โ
โโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโดโโโโโโโโโดโโโโโโโโดโโโโโโโโดโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโ
Highest acceptance: filler (78.1%) Lowest: code (57.4%)
Draft window: 2.1/3 positions used (68% utilization**)** Avg waste: 32%
tool-eval-bench v1.4.3.1
Here is my recipe but itโs really slow to load (>5 minutes) because of trtllm::fused_moe tuner :
# -----------------------------------------------------------------------------
# โ filler @ d0 29.2 eff t/s 29.0 stream t/s ฮฑ=78.1% waste=22% ฯ=2.3 win=3
# โ code @ d0 31.5 eff t/s 31.3 stream t/s ฮฑ=60.1% waste=40% ฯ=1.8 win=3
# โ structured @ d0 31.9 eff t/s 31.7 stream t/s ฮฑ=65.9% waste=34% ฯ=2.0 win=3
# โ filler @ d16384 10.3 eff t/s 10.2 stream t/s ฮฑ=78.1% waste=22% ฯ=2.3 win=3
# โ code @ d16384 30.3 eff t/s 30.1 stream t/s ฮฑ=57.4% waste=43% ฯ=1.7 win=3
# โ structured @ d16384 34.9 eff t/s 34.6 stream t/s ฮฑ=75.2% waste=25% ฯ=2.3 win=3
# โ filler @ d32768 8.9 eff t/s 8.8 stream t/s ฮฑ=69.9% waste=30% ฯ=2.1 win=3
# โ code @ d32768 31.7 eff t/s 31.4 stream t/s ฮฑ=60.7% waste=39% ฯ=1.8 win=3
# โ structured @ d32768 33.9 eff t/s 33.7 stream t/s ฮฑ=70.7% waste=29% ฯ=2.1 win=3
recipe_version: '1'
name: Nemotron Labs 3 Puzzle
description: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
model: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
container: vllm-node
cluster_only: false
solo_only: false
defaults:
host: 0.0.0.0
port: 8002
max_model_len: 131072
max_num_batched_tokens: 8192
num_speculative_tokens: 3
max_num_seqs: 4
gpu_memory_utilization: 0.5 # 1.69x @ ctx=131072 / seq=4
api_key: ""
served_model_name: "Nemotron Labs 3 Puzzle"
env:
VLLM_HTTP_TIMEOUT_KEEP_ALIVE: '1200'
VLLM_FLASHINFER_ALLREDUCE_BACKEND: trtllm
VLLM_ALLOW_LONG_MAX_MODEL_LEN: 1
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: "1"
VLLM_TEST_FORCE_FP8_MARLIN: "1"
VLLM_MARLIN_USE_ATOMIC_ADD: 1
CUDA_VISIBLE_DEVICES: 0
CUTE_DSL_ARCH: "sm_121a"
FLASHINFER_DISABLE_VERSION_CHECK: 1
command: |
vllm serve nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 \
--api-key "{api_key}" \
--host {host} \
--port {port} \
--served-model-name "{served_model_name}" \
--max-model-len {max_model_len} \
--max-num-batched-tokens {max_num_batched_tokens} \
--max-num-seqs {max_num_seqs} \
--gpu-memory-utilization {gpu_memory_utilization} \
--load-format instanttensor \
--async-scheduling \
--enable-prefix-caching \
--enable-chunked-prefill \
--trust-remote-code \
--reasoning-parser nemotron_v3 \
--moe-backend flashinfer_cutlass \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--mamba-backend flashinfer \
--mamba_ssm_cache_dtype float16 \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--kv-cache-dtype fp8 \
--speculative-config '{{"method":"mtp","num_speculative_tokens":3}}' \
--default-chat-template-kwargs '{{"enable_thinking":true}}' \
--override-generation-config '{{"temperature":0.6,"top_p":0.95,"max_tokens":80000}}'
Warning, if you were planning to use it for development spoiler alert: It is unusable (even for basic usage).
damn what a downer โฆ may be we need template fixes?
Hi jbourny โ Iโm running the model (NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4)
on the same class of box (Dell Pro Max / GB10, single Blackwell sm_121, 119 GB unified),
so I can both corroborate your numbers and point at whatโs almost certainly behind your
5-minute load.
Throughput/acceptance โ matches your range. With MTP num_speculative_tokens=3 I get
18.6 โ 28.5 tok/s single-stream (1.53x), per-position draft acceptance 77% / 52% / 36%
(~55% overall). Thatโs ~60% of the practical roofline for this box (~273 GB/s รท ~9B active
@ NVFP4 โ 40-50 tok/s ceiling), so the numbers look healthy โ itโs bandwidth-bound, not a
misconfig.
The load time is the useful bit: your trtllm::fused_moe tuner is the TensorRT-LLM
fused-MoE autotuner running at startup. I donโt hit it at all, because on my build vLLM
auto-selects a different NVFP4 MoE backend. Straight from my engine log:
Using โFLASHINFER_CUTLASSโ NvFp4 MoE backend out of potential backends:
[โFLASHINFER_TRTLLMโ, โFLASHINFER_CUTEDSLโ, โFLASHINFER_CUTEDSL_BATCHEDโ,
โFLASHINFER_CUTLASSโ, โVLLM_CUTLASSโ, โMARLINโ, โEMULATIONโ]
CUTLASS has no multi-minute tuning pass, so my startup (~8 min total) is entirely
safetensors weight-load, not MoE tuning. If your log shows the TRTLLM MoE path (default in
some builds, or if itโs being pinned), steering the NVFP4 MoE backend to FLASHINFER_CUTLASS
should make the tuner disappear. Worth checking which backend your log reports first.
Two more startup notes from my run:
- Weight load dominates. My log says: โAuto-prefetch is disabled because the filesystem
(EXT4) is not a recognized network FS โฆ use --safetensors-load-strategy=prefetch.โ
Adding that trims the 6x~60s shard load on local NVMe. - flashinfer autotune was already off by default on my build
(โSkipping FlashInfer autotune because it is disabledโ, enable_flashinfer_autotune=False)
โ worth confirming yours isnโt switching it on.
On โunusableโ: the load is a one-time cost. I run it under systemd with the container on
โrestart unless-stopped so it stays warm across reboots; after the first load, latency is
just the ~28 tok/s above. If you mean per-token slowness rather than startup, thatโs the
memory-bandwidth ceiling and MTP is the main lever (itโs already working for you at 57-78%
acceptance).
For reference, my stack: nvcr.io/nvidia/vllm:26.06-py3 (vLLM 0.22.1), TP=1, kv-cache
fp8_e4m3, attention backend FLASHINFER, --speculative-config
โ{โmethodโ:โmtpโ,โnum_speculative_tokensโ:3}โ. Happy to compare full configs.
I gave up all nvidia llm models long ago they are terrible nvidia is good with graphics etc not with training llm. Vision stuff is decent hardware too. LLM - nope. at least they try, at some time they will get there too..
You have to think what NVIDIA is, they sell to large companies. They would undercut their own customers if they do SOTA open source models out of the door. I have found that most NVIDIA models have a spesific niche. This one is strategical thinking. Its garbage for coding obviously, and multilanguage leaves much to be desired - but for strategical thinking its beating qwen3.5-122b
Especially in this community where 95% are devs they all tests model in one case โ codding โฆ
Our usecase is marketing, osint and grading.
Tbh nothing really beats the SOTA models for coding. The cost of not using them for defined projects is far higher than their api cost.
But for prospecting, research and idea development itโs often enough with way smaller agents
Thats a nice way to say they build digital consultants - capable of pushing papers, creating useless reports, and telling you how right they are while faking the job done and lying through the teeth. But makes a good excuse for execs to buy more HW. Which is exactly the point. I never seen a more deceptive and hostile model like Nemotron 3 Super. It was faking records, fudging books, and constantly lying. I have everything on record.
Iโm actually getting some excellent results from this model. Itโs fast and is generating some very usable results.
Hereโs my startup script.
cd spark-vllm-docker
./launch-cluster.sh
--solo
-e HF_TOKEN
-e HF_HUB_OFFLINE=1
-e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
-e OMP_NUM_THREADS=4
-e CUDA_MANAGED_FORCE_DEVICE_ALLOC=1
-e TORCH_MATMUL_PRECISION=high
-t vllm-node-tf5:latest
exec vllm serve nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
--tensor-parallel-size 1
--pipeline-parallel-size 1
--data-parallel-size 1
--dtype auto
--enable-prefix-caching
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser nemotron_v3
--gpu-memory-utilization 0.72
--moe-backend cutlass
--mamba-backend flashinfer
--mamba_ssm_cache_dtype float16
--enable-mamba-cache-stochastic-rounding
--mamba-cache-philox-rounds 5
--kv-cache-dtype fp8
--calculate-kv-scales
--max-model-len 262144
--async-scheduling
--speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}'
--override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'
--default-chat-template-kwargs '{"enable_thinking": true, "force_nonempty_content": true}'
--load-format instanttensor
--enable-chunked-prefill
--max-num-batched-tokens 8192
--max-num-seqs 4
--served-model-name nemotron3-puzzle
--host 0.0.0.0
--port 8000
Whats your token per second , and how is the quality ?
I havenโt run a benchmark, but itโs speed is pretty good for agentic work under Hermes.
Itโs not as smart as a larger model like Step 3.7 Flash and tends to have a narrower focus, without considering where the problem lies within a larger context. I think thatโs a common characteristic for most models that can run on a single Spark.
This is basically an official REAP from Nemotron 3 Super 122b. Which is not the strong model for coding and tool calling itself (about 77/100). What is funny, Nemotron Cascade 2 30B A3B is scoring same as Super, but 3 times faster.
The main advantage of Nemotrons - MAMBA attention, that allows tokens to be very small in kv cache (fp8) and long context (1M for both) processed without much speed degradation. But this is no magic - only 2 (AFAIR) attention heads, hence the quality. You always pay with something to gain something else.
thats a big bummer, so even Qwen Coder Next could do better and as fast.
For my use case, I want something thatโs fast and reasonably intelligent. This model seems to provide that. Itโs not for diving into complex software problems. I have another beefier model for that. To be honest, the constant reasoning loops that you get with the Qwen models makes me avoid them. I donโt want to be baby-sitting the agent just so that I can get the model back on track every 5 minutes.
Anyway, I asked the model what it thought of the discussion in this thread. Hereโs an extract -
Benchmarks vs. real-world utility:
Benchmark scores (e.g., MMLU, GSM8K, etc.) often emphasize broad knowledge and reasoning breadth. For agentic workflowsโwhere the model is integrated into a tool-using, goal-directed loopโfactors like response speed, consistency in tool use, and reliability on specific task patterns may matter more than peak benchmark performance. A faster, slightly less โknowledgeableโ model can sometimes be more effective in a well-designed agent loop than a slower, more knowledgeable one that stalls on tool selection or reasoning loops.
Conclusion:
The model isnโt โuselessโโitโs optimized for a different operating regime. If your workflow values low latency and local execution, its performance may be perfectly suitable. If you find yourself frequently hitting its limits on complex reasoning or niche knowledge, you might consider a larger model for those specific tasksโbut for many agentic scenarios, the speed/respect trade-off can be worthwhile.
The bench we used is not measuring reasoning - this is really individual task, it measures ability of tool-calling, exactly suitability for agentic tasks. And Nemotrons score pretty lousy at it. But if you need speed above everything - check out Nemotron Cascade 2 โ 60 t/s on a single spark, 1M session, 5M cache.