Hey guys just landing this here and will update as I go along. Feel free to send issues and PRs.
From Agent -
Qwen3.8-Flash-Next NVFP4 on 2x DGX Spark — SGLang TP2, MTP4, CUDA graphs, full multimodal: 70 tok/s peak / 47 typical
Since a few folks asked about the 2-Spark run mentioned earlier in the thread — here’s the full recipe, including the two one-line patches that make it work on GB10/SM121 and the day-0 bugs we hit so you don’t have to.
Setup
- 2x DGX Spark (GB10, SM121), ConnectX fabric between them (RoCE/IB)
- Checkpoint:
RadixArk/Qwen3.8-Flash-Next-NVFP4(135 GB — NVFP4 experts + FP8 n-gram table, vision tower left BF16). One copy on the head node, NFS-shared to the worker. ~63-76 GB resident per rank. - Engine: SGLang day-0 image
lmsysorg/sglang:qwen38flashnext. vLLM won’t load this arch — we probed two vLLM images and neither registersQwen4ExpForConditionalGeneration. SGLang nightlies don’t have it either (support PR sgl-project/sglang#36497 is still unmerged), so you’re pinned to the day-0 tag for now.
Patch 1 — SM121 QSA guard (needed to boot at all)
The image gates its FlashInfer TRT-LLM sparse-decode kernel to SM100 and falls back to an FA4 CUTE path on SM121 — which dies in warmup with an MLIR “weakly congruent” error. The bundled TRT-LLM kernel actually passes a direct probe on SM121 with this model’s head shape (24 q-heads / 2 kv-heads / head_dim 256), so the fix is extending the guard:
# python/sglang/srt/layers/attention/qwen_sparse_attn_backend.py
- if not is_sm100_supported():
+ if not (is_sm100_supported() or is_sm120_supported()):
return None
Patch 2 — mrope partial-rotary OOB fix (needed for stable vision)
This one’s subtle and, as far as we can tell, unreported anywhere: qwen4_exp is the first M-RoPE model with partial rotary (partial_rotary_factor 0.25 → rotary width 64 on head_dim 256, so the cos/sin cache row is 32 half-lanes). The fused mrope Triton kernel pads its lane range to next_power_of_2(head_dim)//2 = 128, and its temporal mask t_mask = ~(h_mask | w_mask) is unbounded above the cache row width — every earlier mrope model (Qwen2.5/3-VL) had full-width rotary, so the masks were incidentally bounded and nobody noticed. Result: ~96 lanes of out-of-bounds cache reads on every image call, walking off the end of the cache near the 262K context edge. Under captured CUDA graphs we saw a rare device-side assert (~once per 90 min of vision load) surfacing in _compute_mrope_positions_extend. The bound is one line and provably a no-op for full-rotary models:
# python/sglang/kernels/ops/attention/rotary_triton.py (interleaved branch)
- t_mask = ~(h_mask | w_mask)
+ t_mask = ~(h_mask | w_mask) & (cos_offsets < half_rd)
(The neighboring w_mask on the very next line was already bounded — that asymmetry is the tell.) Related upstream work: sgl-project/sglang#36021 bypasses the fused path for multi-dim positions on the Qwen3.5 family; still open.
Serve config (TP2, worker rank first, then head)
sglang serve --model-path /models/qwen3.8-flash-next-nvfp4 \
--tp-size 2 --nnodes 2 --node-rank <0|1> --dist-init-addr <head-fabric-ip>:29511 \
--quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass \
--page-size 64 --mamba-scheduler-strategy extra_buffer --mamba-track-interval 64 \
--speculative-algorithm NEXTN --speculative-num-steps 3 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
--enable-linear-replayssm-spec \
--chunked-prefill-size 4096 --max-running-requests 6 --context-length 262144 \
--max-total-tokens 600000 --mem-fraction-static 0.80 \
--reasoning-parser auto --tool-call-parser qwen3_coder \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--ple-offload-embedding --cuda-graph-max-bs 8 --disable-cuda-graph-padding \
--disable-radix-cache --sampling-backend pytorch --trust-remote-code
The MTP4 spec decode uses the checkpoint’s built-in MTP layer — no separate draft model.
Numbers (single-stream, 200-token decode)
| Config | tok/s |
|---|---|
| No speculative | 20 |
| + MTP4 (NEXTN 3 steps / 4 draft) | 33 typical, 55 peak warmed |
| + CUDA graphs (decode bs≤8) | 47 typical, 70.2 peak |
TTFT ~0.2-0.7 s. Acceptance 0.36-0.56 cold, better on code. ~6 min to SERVING (206 shards, ~76 GB/rank).
Day-0 bugs you will hit (and the fixes)
- Token-0
!!!!!loops in agentic use: thinking mode + OpenAItools+--tool-call-parser qwen3_codermakes the model loop on token ID 0 (decodes as!) until max_tokens — sgl-project/sglang#36537. Fix:--default-chat-template-kwargs '{"enable_thinking": false}'server-side (tool calls stay structured), and keep request temperature ≤ 0.7 — there’s a residual loop regime at temp 1.0 even with thinking off. - GB10 unified-memory OOM under load: at mem-fraction 0.82 the KV pool + OS page cache starve the GPU allocator (
NV_ERR_NO_MEMORY) —free -glies. Fix: pin with--max-total-tokens 600000 --mem-fraction-static 0.80, andsync; echo 3 > /proc/sys/vm/drop_cacheson both nodes before every launch, non-negotiable. - Multi-node relaunch trap:
docker rm -fBOTH containers before starting either rank. If the old head is still up when the new worker starts, the worker rendezvouses with the dying head, exits 0 (looks clean!), and the new head hangs forever at “Init torch distributed begin”. max_tokensincludes hidden reasoning tokens if you ever re-enable thinking — tiny caps return empty content.
Vision status: with patch 2 + graphs on we’re running full multimodal at the 70-class config as of tonight; the pre-patch assert was ~1/90 min so we’re soaking it — will report back. If you need bulletproof today: add --disable-cuda-graph (~55 tok/s peak, vision on, zero asserts in our testing).
Full recipe, launcher scripts, patches, benchmarks, and the incident log:
https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-2x-DGX-Spark