Firstly, a big thanks to @stu.miller and @0rand for the pointers that got me here. To set the stage here, my goal is to have a 2-node Spark cluster for agentic-driven coding by a small team, maybe 5-10 users (ok, 10 is probably stretching it…)
TL;DR — I got aidendle94/sparkrun-vllm-ds4-gb10:production-3.7 (DSpark checkpoint,
NVFP4/b12x) running well across two DGX Sparks (GB10, sm_121, TP=2 over 200 Gb/s CX-7 RoCE).
After a full config sweep the single biggest lever was not overriding the image’s
max_cudagraph_capture_size=128 — and, on GB10, num_speculative_tokens=3 beats the
image default of 5. Numbers and a copy-paste config below.
Hardware / stack
- 2× HP ZGX Nano’s (DGX Spark GB10 “clones”), 128 GB unified each, TP=2 over a direct-cabled ConnectX-7 (RoCEv2, ~112 Gb/s).
- Image:
aidendle94/sparkrun-vllm-ds4-gb10:production-3.7 (vLLM …eldritch…b12x…), checkpoint deepseek-ai/DeepSeek-V4-Flash-DSpark.
- Backends confirmed active in boot log:
b12x MoE, FLASHINFER_MLA_SPARSE_DSV4 attention,
dspark speculative, DeepGEMM E8M0/PDL.
The five things that actually mattered (reproduction notes)
-
Don’t shrink max_cudagraph_capture_size (image default 128). This was our misstep — we’d set it to 8. Cudagraphs then only cover batch ≤8, so every larger batch runs eager and concurrency collapses. Restoring 128 was worth +40–170% aggregate throughput at concurrency. If you override image env, leave this (and max_num_seqs=128) alone.
-
num_speculative_tokens=3 > 5 on GB10. The image’s default is 5. We measured spec=5’s per-position acceptance collapsing at the tail (positions 4–5 accept ~0.36/0.22), so it does ~67% more draft+verify compute for ~17% more accepted tokens — slower single-stream and slightly slower under load. spec=3 won on both. (Your mileage may vary by draft quality, but worth an A/B test.)
-
Warm up before you benchmark. First ~2 min after boot are ~30% slow — cold DeepGEMM JIT + FlashInfer autotune. Fire ~30 warm-up requests first or you’ll under-report by a third. (This alone explained an early “b12x isn’t engaging” scare — it was engaged; the kernels were cold.)
-
Fabric (direct-docker only). If you run the container by hand (not via sparkrun): the image bakes socket ifnames to an f1 port (enp1s0f1np1) — override GLOO/NCCL/TP_SOCKET_IFNAME to your wired rail. And the RoCE-v2 IPv4 GID index is per-node (ours: 3 on node0, 4 on node1 — show_gids on both, don’t assume symmetry). Under sparkrun you can skip all this — sparkrun sets up the 2-node NCCL/RoCE fabric for you.
-
max_model_len + static YARN (thanks Orand). DS4F ships a static YARN pinned to 1M. Lowering --max-model-len is fine, but check the vLLM boot line GPU KV cache size: … tokens / Maximum concurrency: …x — some length values cost you KV pool. On my box 384K actually reported more usable KV (≈2.05M tok, 5.2×) than 500K (≈1.53M, 3.1×), because the smaller max_num_batched_tokens=4096 leaves more memory for KV. Pick length by watching that line.
(Aside: the checkpoint “revision drift” is a non-issue — I diffed HF main vs the pinned 913f0657… blob-by-blob; identical except README.md.)
Metrics (warm, TP=2, short-prompt unless noted; aggregate tokens/s)
|
single-stream |
conc-4 |
conc-8 |
conc-16 |
deep single @108K |
tool-calls |
| hybrid-v3 (what we run) |
~43–48 |
111 |
153 |
242 |
43 |
3/3 |
| container-exact (spec=5) |
37 |
93 |
152 |
201 |
40 |
3/3 |
- Single-stream tops out ~48–51 t/s (vLLM’s own
Avg generation throughput log line reads ~51).
- Mean spec acceptance length ~3.0; KV pool ≈2.05M tokens → 5.2× concurrency at 384K.
- Reproduces (and slightly exceeds) the “≈80 t/s at c4” figure people quote — as aggregate at concurrency 4.
The config (image defaults + 4 changes)
Keep the image env defaults (SPEC_TOKENS=5→ I set 3, MAX_NUM_BATCHED_TOKENS=4096,
MAX_NUM_SEQS=128, GRAPH_CAP=128, GPU_MEM=0.80→ I use 0.85) and change only the serve line:
--max-model-len 393216 # 384K (check the KV boot line)
--default-chat-template-kwargs.reasoning_effort=high # image ships 'max'; 'high' scored better on tool-eval
--override-generation-config '{"temperature":0.4}'
--speculative-config '{"method":"dspark","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
Everything else (attention FLASHINFER_MLA_SPARSE_DSV4, --moe-backend b12x, fp8 KV,
--enable-flashinfer-autotune, the deepseek_v4 parsers) is the image’s own serve command.
Extra credit — sparkrun recipe (validated, no perf regression)
A recipe_version: '2' recipe for the sparkrun path — validated on our cluster: serves cleanly
and benchmarks within noise of direct-docker (conc-8 154 vs 153, conc-16 236 vs 242, single ~42,
KV 2.04M/5.19×, tools 3/3). sparkrun handles the 2-node orchestration + NCCL/RoCE fabric, so you
skip the socket-ifname + per-node-GID gotchas entirely. Full file:
bench/deepseek/dspark-hybrid-v3.sparkrun.yaml.
One gotcha you MUST handle: sparkrun runs the container as a non-root UID (1000), but the
aidendle image points its JIT caches at root-owned /cache/jit — so the worker dies with
PermissionError: /cache/jit/... unless you redirect them. The env: block below points all six
(TRITON_/TORCH_EXTENSIONS_/VLLM_CACHE_DIR, FLASHINFER_WORKSPACE_BASE, TVM_FFI_CACHE_DIR, XDG_CACHE_HOME) into a subdir of the mounted HF cache (/cache/huggingface/jitcache) — writable by the non-root UID and persistent, so compiled kernels survive restarts (the image ships no prebaked artifacts — /cache/jit is empty — so nothing is lost). Also run sparkrun setup fix-permissions for the HF cache.
Key bits:
recipe_version: '2'
model: deepseek-ai/DeepSeek-V4-Flash-DSpark
runtime: vllm
min_nodes: 2
container: aidendle94/sparkrun-vllm-ds4-gb10:production-3.7
defaults:
tensor_parallel: 2
gpu_memory_utilization: 0.85
max_model_len: 393216
max_num_seqs: 128
max_num_batched_tokens: 4096
max_cudagraph_capture_size: 128 # ⭐ leave at 128
kv_cache_dtype: fp8
tokenizer_mode: deepseek_v4
tool_call_parser: deepseek_v4
reasoning_parser: deepseek_v4
speculative_config: '{"method":"dspark","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
compilation_config: '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
gen_config: '{"temperature":0.4}'
env:
VLLM_ENABLE_PCIE_ALLREDUCE: '0'
VLLM_PCIE_ALLREDUCE_BACKEND: 'cpp'
TRITON_CACHE_DIR: /cache/huggingface/jitcache/triton # off root-owned /cache/jit; persistent
TORCH_EXTENSIONS_DIR: /cache/huggingface/jitcache/torch_extensions
VLLM_CACHE_DIR: /cache/huggingface/jitcache/vllm
FLASHINFER_WORKSPACE_BASE: /cache/huggingface/jitcache/flashinfer
TVM_FFI_CACHE_DIR: /cache/huggingface/jitcache/tvm-ffi
XDG_CACHE_HOME: /cache/huggingface/jitcache
command: |
vllm serve {model} --host {host} --port {port} --trust-remote-code \
-tp {tensor_parallel} --kv-cache-dtype {kv_cache_dtype} --block-size 256 --load-format auto \
--gpu-memory-utilization {gpu_memory_utilization} --max-model-len {max_model_len} \
--max-num-seqs {max_num_seqs} --max-num-batched-tokens {max_num_batched_tokens} \
--max-cudagraph-capture-size {max_cudagraph_capture_size} \
--compilation-config '{compilation_config}' --async-scheduling --no-scheduler-reserve-full-isl \
--enable-chunked-prefill --enable-prefix-caching --enable-flashinfer-autotune \
--tokenizer-mode {tokenizer_mode} --tool-call-parser {tool_call_parser} --reasoning-parser {reasoning_parser} \
--enable-auto-tool-choice --override-generation-config '{gen_config}' \
--default-chat-template-kwargs.thinking=true --default-chat-template-kwargs.reasoning_effort=high \
--attention-backend FLASHINFER_MLA_SPARSE_DSV4 --moe-backend b12x --disable-custom-all-reduce \
--speculative-config '{speculative_config}'
sparkrun run dspark-hybrid-v3.sparkrun.yaml --cluster <your-2-node-cluster>
Both paths (direct-docker and this sparkrun recipe) land at the same performance — pick whichever
fits your ops. If you’re already a sparkrun shop, the recipe is the tidier route (no manual fabric
setup). Happy to compare notes / .envs.