GLM-5.3-Flash (320B/18B MoE, LibertAIDAI NVFP4 quant) with the incoai DFlash2 block-diffusion drafter, running on the upstream SGLang path (PR #36507 branch) across 2x DGX Spark, TP=2. As far as we can tell this is the first published SGLang-route deployment of this pairing — @tonyd615 got the vLLM route running the same night (his numbers are faster; credit where due, and his GB10 forensics carried half our bring-up).
29.4 tok/s code / 23.4 prose decode with DFlash2, vs 14.7 no-spec on the same stack (1.88x / 1.41x)
accept length ~5-6 of 6-token blocks on code; TTFT ~2.3 s at 4K prompt, 7.9 s at 16K
production envelope: 131K context, bf16 KV (fp8 KV is currently impossible on SGLang GB10 — measured backend matrix in the repo and in sglang issue #36830)
Concurrency curve (agent-style prompts, 400 tok, measured today — spec decode and throughput want DIFFERENT boots):
DFlash2 on: 35.5 tok/s at c1, but it saturates the machine there — c2 adds ~nothing
no-spec, 12 streams: 14.5 (c1) → 27.1 (c2) → 36.8 (c4) → 55.1 (c8) → 55.0 (c12) tok/s aggregate — the ~55 plateau is the bandwidth ceiling for this model on 2 Sparks
practical takeaway: DFlash2 for interactive/latency, no-spec for agent fleets
Four GB10 day-0 fixes you will hit if you try this (all patched + probed in the repo):
SGLANG_HOST_IP must be set per-rank or multi-node shm_broadcast hangs forever
DFLASH x hybrid-KDA memory law: the mamba pool needs per_req*(1+D) plus ~5x per-request amplification — drafter-less memory configs fail with max_mamba_cache_size<=0
DSA tilelang smem overflow at the 8-token verify shape (169,984 B > 101,376 B) — retune to block_I=32 / num_stages=1 / threads=128. We mapped the whole tile space: fat 64-wide tiles are physically impossible on GB10 (>=103.4 KB in any stage/thread shape), and the small tile costs nothing measurable
residual=None crash in the DFLASH capture adapter — since fixed upstream (sglang #36755)
Honesty section: DFlash2 on this stack is NOT bit-identical to spec-off at temp 0 (1/20 exact in a 20-prompt matrix with reasoning captured; quality preserved, exactness not — reported to incoai). Every number above names its prompt, token count, and clock state; the repo ladder includes the reverted experiments, not just the kept ones.
Next up from this rig: a tilelang fp8-KV path for CUDA (the biggest single lever — see sglang #36830) and a decoupled-drafter experiment.
Update, ~1 hour later: full concurrency curves are in, and they correct one line in my original post.
I wrote that DFlash2 “saturates the machine at c1 concurrency adds ~nothing.” That turned out to be an artifact of --max-running-requests 2. With headroom (max-running-requests 8), DFlash2 scales:
DFlash2 dominates the whole curve until ~c8 — at c4 it delivers 3x the per-stream speed of no-spec at nearly the same aggregate
~55 tok/s aggregate is the bandwidth plateau for this model on 2 Sparks regardless of config
Our production boot is now DFlash2 + max-running-requests 8 + 131K context: 37 tok/s single-stream, ~50 aggregate under fleet load, one config for both interactive and agent traffic
Next from this rig: attempting a tilelang fp8-KV path for CUDA (currently impossible on SGLang GB10 — matrix in sglang #36830). That’s the biggest single lever left toward the vLLM route’s 46.9.
Big update since the original post. Three “known limitations” of the SGLang route died in one day, then we ran the first same-rig comparison against the EXL3+vLLM recipe.
fp8 KV cache on GB10 works. The tilelang tree already ships a complete raw-fp8 sparse kernel, HIP-gated in three plumbing sites. Ported, validated on sm_121 (decode parity with bf16, 4/5 temp-0 exact, 32k recall clean, TTFT@16k 6.6s vs 7.9s). Upstream PR: sglang #36904.
The ~55 tok/s “bandwidth plateau” was a silent cap: the mamba state cache limits EVERY DFlash config to 2 concurrent streams (boot log: “capped to 2 by the mamba state cache”). Fix: --max-mamba-cache-size 40 --mamba-ssm-dtype bfloat16 --mem-fraction-static 0.92. Result: c8 = 78 tok/s aggregate (8/8 truly concurrent), c12 = 83.5 vs 48.8 capped. Filed as sglang #36889. We also ran a correctness matrix under load (fixed arithmetic, temp 0): 44/44 right answers at c1/c4/c8.
GLM-5.3-Flash is multimodal and the SGLang path serves it: --enable-multimodal. Image input verified working on the fp8 + 8-stream config, no measurable decode tax.
THE SHOOTOUT: we then ran MiaAI-Lab’s EXL3+vLLM recipe (their repo @ bd7f55e, their image, their defaults) on the SAME two Sparks, same prompts, same protocol. Honest split:
EXL3+vLLM wins single-user code decisively: 51.9 vs 29.3 tok/s
SGLang fp8 wins prose (29.2 vs 23.7) and fleet concurrency: 83.5 vs 53.3 at c12 (+57%); the EXL3 lane’s aggregate degrades past c4.
Solo coder: run their lane. Agents/teams/concurrent: run this one. Both live on our rig now; we re-measure as they ship.
One warning for long-context users on EITHER stack: we can reproduce a silent worker-node death on long prefills (~62k tokens killed rank1 with no traceback on a 262k-context config; 32k passes). Threshold bisect in progress; treat >32k prompts as unverified until we post the follow-up. Everything above, with the full ladder including failed experiments: