I spent a week benchmarking LLM inference on a single DGX Spark and published everything, including the mistakes. Sharing here in case it saves someone time, and because I would like other Spark owners to check my numbers against theirs.
Method: identical prompt, 400 max_tokens, warm-up request discarded, then single-stream, 8 concurrent, and a long-prompt prefill measurement. Same harness for every stack — that turned out to matter more than anything else I did.
Same model, both stacks, one harness
| Model | Ollama Q4 | vLLM 4-bit | Ollama 8x | vLLM 8x | Ollama TTFT | vLLM TTFT |
|---|---|---|---|---|---|---|
| qwen3-coder-next | 59.2 | 74.8 | 60.4 | 289.0 | 215 ms | 104 ms |
| Gemma-4-26B-A4B + MTP | 49.6 | 54.9 | 51.9 | 303.6 | 511 ms | 94 ms |
| GPT-OSS-120B | 42.1 | 60.7 | 42.5 | 116-153 | 449 ms | 131 ms |
| Qwen3-30B-A3B | 85.2 | 77.1 | 87.5 | 313.4 | 161 ms | — |
Ollama’s aggregate at 8 concurrent equals its single-stream figure on all four models — it does not batch. vLLM reaches 289-313 tok/s on the same hardware.
Things that may be useful to others here
NVFP4 is not a trap, missing kernels are. The same weights ran at ~1.1 tok/s on vanilla vLLM (emulation fallback) and 77.1 tok/s with FlashInfer Cutlass kernels. SM121 lacking cvt.e2m1x2 removes the FP4 compute speedup but not the bandwidth saving, and bandwidth is the binding constraint on this box. Worth checking the log line rather than the format name.
Speculative decoding pays off unusually well here. On Gemma-4-26B-A4B-NVFP4, MTP took single-stream from 30.3 to 54.9 tok/s (+81%) — a bigger gain than any quantization change I tried. That makes sense on bandwidth-bound hardware: verification is nearly free when the weights had to be read anyway. Also: num_speculative_tokens: 2 beat 4, because per-position acceptance decayed 0.84 → 0.60 → 0.39 → 0.27.
Prefill is not the bottleneck. ~6,000 tok/s prefill against ~56 tok/s generation, flat out to 27k tokens. A 27,000-token prompt prefills in 3.6 s. For RAG the context is nearly free.
Tool-calling failures were usually the server, not the model. Three models I had written off as unusable for agents work fine under vLLM with matching --tool-call-parser / --reasoning-parser.
One observation on NVIDIA-published NVFP4 checkpoints
nvidia/Gemma-4-26B-A4B-NVFP4 measured 30.3 tok/s single-stream without speculative decoding, against 49.6 tok/s for a community Q4_K_M GGUF of the same model in Ollama. With MTP enabled it moves to 54.9. If that gap is expected, it might be worth a note on the model card, since the obvious A/B makes the NVFP4 checkpoint look slower than it is.
I also see this at startup on every vLLM run, which I assume is known:
SM12x detected - using native FlashInfer CUTLASS attention
instead of TRT-LLM attention (cubins not available for SM12x)
Caveats
One machine, one operator, one prompt set. Quality was not measured at all — only speed. Concurrent-throughput figures moved ±30% between runs on identical configurations (prefix-cache warming), so I would not lean on that column. Everything is pinned to specific versions in the repo, because vLLM nightlies move fast enough that a week matters.
If you have a Spark and your numbers disagree with mine, I would genuinely like to know — there is an issue template for exactly that.
Environment: DGX Spark (GB10), 121 GiB unified, Ubuntu 24.04 ARM64, driver 580.173.02, CUDA 13.0, vLLM 0.26.1rc1.dev535+g83ad767ee.