Qwen3.8-Flash-Next NVFP4 on 2x DGX Spark full multimodal: 70 tok/s peak / 47 typical

Hey guys just landing this here and will update as I go along. Feel free to send issues and PRs.

From Agent -

Qwen3.8-Flash-Next NVFP4 on 2x DGX Spark — SGLang TP2, MTP4, CUDA graphs, full multimodal: 70 tok/s peak / 47 typical

Since a few folks asked about the 2-Spark run mentioned earlier in the thread — here’s the full recipe, including the two one-line patches that make it work on GB10/SM121 and the day-0 bugs we hit so you don’t have to.

Setup

  • 2x DGX Spark (GB10, SM121), ConnectX fabric between them (RoCE/IB)
  • Checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4 (135 GB — NVFP4 experts + FP8 n-gram table, vision tower left BF16). One copy on the head node, NFS-shared to the worker. ~63-76 GB resident per rank.
  • Engine: SGLang day-0 image lmsysorg/sglang:qwen38flashnext. vLLM won’t load this arch — we probed two vLLM images and neither registers Qwen4ExpForConditionalGeneration. SGLang nightlies don’t have it either (support PR sgl-project/sglang#36497 is still unmerged), so you’re pinned to the day-0 tag for now.

Patch 1 — SM121 QSA guard (needed to boot at all)

The image gates its FlashInfer TRT-LLM sparse-decode kernel to SM100 and falls back to an FA4 CUTE path on SM121 — which dies in warmup with an MLIR “weakly congruent” error. The bundled TRT-LLM kernel actually passes a direct probe on SM121 with this model’s head shape (24 q-heads / 2 kv-heads / head_dim 256), so the fix is extending the guard:

# python/sglang/srt/layers/attention/qwen_sparse_attn_backend.py
- if not is_sm100_supported():
+ if not (is_sm100_supported() or is_sm120_supported()):
      return None

Patch 2 — mrope partial-rotary OOB fix (needed for stable vision)

This one’s subtle and, as far as we can tell, unreported anywhere: qwen4_exp is the first M-RoPE model with partial rotary (partial_rotary_factor 0.25 → rotary width 64 on head_dim 256, so the cos/sin cache row is 32 half-lanes). The fused mrope Triton kernel pads its lane range to next_power_of_2(head_dim)//2 = 128, and its temporal mask t_mask = ~(h_mask | w_mask) is unbounded above the cache row width — every earlier mrope model (Qwen2.5/3-VL) had full-width rotary, so the masks were incidentally bounded and nobody noticed. Result: ~96 lanes of out-of-bounds cache reads on every image call, walking off the end of the cache near the 262K context edge. Under captured CUDA graphs we saw a rare device-side assert (~once per 90 min of vision load) surfacing in _compute_mrope_positions_extend. The bound is one line and provably a no-op for full-rotary models:

# python/sglang/kernels/ops/attention/rotary_triton.py (interleaved branch)
- t_mask = ~(h_mask | w_mask)
+ t_mask = ~(h_mask | w_mask) & (cos_offsets < half_rd)

(The neighboring w_mask on the very next line was already bounded — that asymmetry is the tell.) Related upstream work: sgl-project/sglang#36021 bypasses the fused path for multi-dim positions on the Qwen3.5 family; still open.

Serve config (TP2, worker rank first, then head)

sglang serve --model-path /models/qwen3.8-flash-next-nvfp4 \
  --tp-size 2 --nnodes 2 --node-rank <0|1> --dist-init-addr <head-fabric-ip>:29511 \
  --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass \
  --page-size 64 --mamba-scheduler-strategy extra_buffer --mamba-track-interval 64 \
  --speculative-algorithm NEXTN --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --enable-linear-replayssm-spec \
  --chunked-prefill-size 4096 --max-running-requests 6 --context-length 262144 \
  --max-total-tokens 600000 --mem-fraction-static 0.80 \
  --reasoning-parser auto --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --ple-offload-embedding --cuda-graph-max-bs 8 --disable-cuda-graph-padding \
  --disable-radix-cache --sampling-backend pytorch --trust-remote-code

The MTP4 spec decode uses the checkpoint’s built-in MTP layer — no separate draft model.

Numbers (single-stream, 200-token decode)

Config tok/s
No speculative 20
+ MTP4 (NEXTN 3 steps / 4 draft) 33 typical, 55 peak warmed
+ CUDA graphs (decode bs≤8) 47 typical, 70.2 peak

TTFT ~0.2-0.7 s. Acceptance 0.36-0.56 cold, better on code. ~6 min to SERVING (206 shards, ~76 GB/rank).

Day-0 bugs you will hit (and the fixes)

  1. Token-0 !!!!! loops in agentic use: thinking mode + OpenAI tools + --tool-call-parser qwen3_coder makes the model loop on token ID 0 (decodes as !) until max_tokens — sgl-project/sglang#36537. Fix: --default-chat-template-kwargs '{"enable_thinking": false}' server-side (tool calls stay structured), and keep request temperature ≤ 0.7 — there’s a residual loop regime at temp 1.0 even with thinking off.
  2. GB10 unified-memory OOM under load: at mem-fraction 0.82 the KV pool + OS page cache starve the GPU allocator (NV_ERR_NO_MEMORY) — free -g lies. Fix: pin with --max-total-tokens 600000 --mem-fraction-static 0.80, and sync; echo 3 > /proc/sys/vm/drop_caches on both nodes before every launch, non-negotiable.
  3. Multi-node relaunch trap: docker rm -f BOTH containers before starting either rank. If the old head is still up when the new worker starts, the worker rendezvouses with the dying head, exits 0 (looks clean!), and the new head hangs forever at “Init torch distributed begin”.
  4. max_tokens includes hidden reasoning tokens if you ever re-enable thinking — tiny caps return empty content.

Vision status: with patch 2 + graphs on we’re running full multimodal at the 70-class config as of tonight; the pre-patch assert was ~1/90 min so we’re soaking it — will report back. If you need bulletproof today: add --disable-cuda-graph (~55 tok/s peak, vision on, zero asserts in our testing).

Full recipe, launcher scripts, patches, benchmarks, and the incident log:

https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-2x-DGX-Spark

Repo link seems to be

thank you