GLM-5.3-Flash on 4× DGX Spark: ~30–43 tok/s, 1M context, uncensored, multimodal
Native FP8. Tensor-parallel 4. CUDA graphs on. Stock vLLM does not boot this model on GB10. This is the recipe that does.
Repo (setup, lanes, benches, patches): GitHub - Wpnx330/GLM-5.3-Flash-FP8-4x-DGX-Spark: GLM-5.3-Flash native FP8 on 4x DGX Spark (GB10): occupancy lanes 15@200k / 5@500k / 3@1M, CUDA graphs, multimodal. Followable recipe. · GitHub
Tony’s 2-Spark NVFP4 writeup is what made GB10 even possible. We took his SM121 image layers, kept FP8 + DeepGEMM, and ran occupancy lanes on four boxes.
Numbers (2026-08-27)
4× Spark, GB10, gmu=0.85, MTP k=3, graphs + async scheduling. Streaming /v1/chat/completions. TTFT = first content token. Large prompts uniquely salted so prefix cache cannot cheat.
| Workload | Prefill | Decode | TTFT | E2E |
|---|---|---|---|---|
| Tiny ping (~20 tok) | — | 28–34 tok/s | 0.31–0.33 s | ~1.3 s |
| ~50k unique prompt | ~2,100 tok/s | ~43 tok/s | ~25 s | ~25 s |
| 1280×640 PNG (1085 vision tok) | encoder + 150–700 tok/s | 32–39 tok/s | ~1.5 s warm / ~7 s cold | ~6–11 s |
Five concurrent ~50k unique prefills: per-stream decode ~8–10 tok/s, TTFT ~75 s. That is HBM sharing.
Short-chat speed is the same on the 200k, 500k, and 1M lanes. The lane is a KV / seq cap, not a speed tax on a 20-token ping.
Tiny-prompt E2E was ~2.4 s with --enforce-eager and ~1.3 s after graphs captured.
We did not run 15-wide. Do not read 15 @ 200K as 15 × 30 tok/s.
Thinking-on decode tok/s is inflated (thinking tokens land in completion_tokens after the first content token). Quote E2E / TTFT for those rows.
Full tables: docs/BENCH.md
Lanes
One engine. Restart to switch. Default gmu=0.85.
| Lane | Context | Seq slots | When |
|---|---|---|---|
200k |
200,000 | 15 | Many short and medium chats |
500k |
500,000 | 5 | Daily. Agent sessions that hold a repo |
1m |
1,000,000 | 3 | Long dumps |
These seq counts are occupancy. They are not a promise that 15 full 200K windows fit in the 0.85 KV pool. Engine print is ~2.1–2.4M KV tokens: about 11× at 200K, 4.7× at 500K, 2.15× at 1M. Mixed chats are what 15/5/3 are sized for. Dump every slot at max length and you get preemption.
If you stripped the desktop, GLM53_GMU=0.885 is the squeeze we ran on GLM-5.2. Leave 0.85 if the GUI is still there. Rank 0 also holds the API.
One file configures a foreign cluster: scripts/cluster.env (copy from cluster.env.example). IPs, SSH user, NCCL devices. Scripts do not hardcode a fabric.
How we tested
Four scenarios × concurrency 1–5, on each lane:
- Tiny ping, thinking off
- Short reasoning prompt, thinking on
- ~50k unique pad
- Pharos CLI PNG via
image_url(1085 prompt tokens every run)
Video was checked separately (17s 1080p MP4; named the CLI and the commands). Not in the 1–5 matrix.
What had to change
Stock image dies after 62/62 shards. SM120 sparse MLA hard-wires fp8_ds_mla and wants pe_dim=64. GLM-5.3 is NoPE (qk_rope_head_dim=0). Other backends: compute capability not supported. Tony’s Dockerfiles extend the SM90 NoPE backend to SM121, bump FlashInfer to 0.6.18, re-pin NCCL 2.30.7 and cutlass-dsl 4.6.2.
--kv-cache-dtype fp8 selected fp8_ds_mla. We serve fp8_e4m3.
Official ENTRYPOINT is already vllm serve. Passing it again exits immediately.
CUDA graphs. Tony’s 5.3 writeup stays on --enforce-eager. Dropping it here hit persistent_topk wanting 128 KB SMEM. GB10 has 101 KB. Same class of bug as his fp8-KV CTA tile. Bind-mounts skip that kernel on SM12x. Graphs then captured (PIECEWISE 8/8, FULL 3/3). --async-scheduling stays on.
FP8, not NVFP4. NVFP4 on GB10 is Marlin dequant. Usually slower decode. Same lesson as GLM-5.2.
No DCP. 5.2 needed it. This card does not.
MTP k=3, batched tokens 8192, --block-size 2304.
Longer version: docs/DESIGN.md
Two Sparks
This recipe is TP=4 because native FP8 is ~78 GiB of weights per rank. Two boxes means TP=2 and almost no KV. Tony’s NVFP4 + Marlin path is the working 2-Spark recipe (262K). We did not ship 2-Spark occupancy files for this FP8 card.
Follow it
Weights: dealignai/GLM-5.3-Flash-UNCENSORED-FP8
SM121 image layers: tonyd2wild/GLM-5.3-Flash-NVFP4-262K-2x-DGX-Spark
