I can’t believe we have so many great models and recipes to try out! I was super happy with @stephen.douglas.hawkins’s DGPP’s very fast implementation of Qwen on 2xsparks! Then onto @christopher_owen’s 3xspark switchless DS4.1F … better coding output for me than Qwen, and faster! … who said you can’t solve a problem by throwing more hardware at it? :) … and now this new GLM update! Just a fantastic job by the community here, you are all just amazing. Thank you!
LOVE THIS THANK YOU
GLM-5.3-Flash NVFP4 (NVIDIA checkpoint) on 2× DGX Spark — TP2 + DCP2, 512K, +23-27% decode with online NVFP4 of the BF16 dense layers
Recipe + mods are up as a PR on eugr’s repo: GLM-5.3-Flash NVIDIA NVFP4 on 2× DGX Spark: TP2+DCP2, MTP k3, 512K, online NVFP4 of the BF16 dense layers (+23-27% decode) by vedcsolution · Pull Request #421 · eugr/spark-vllm-docker · GitHub
The finding. nvidia/GLM-5.3-Flash-NVFP4 (rev 09b04e5e) keeps ~11.3 GiB of attention projections (KDA q/k/v/o) and ~2 GiB of shared experts in BF16. At TP2 that’s ~6.7 GiB per rank read on every decode step — about half the bandwidth floor of a step on GB10. A mod quantizes them at load time with the fork’s own online methods (MXFP8 or NVFP4 W4A16). The checkpoint on disk is untouched; MTP acceptance doesn’t change.
Results (2× Spark, DGX OS 7.6 / kernel 7.0, GPU clock capped at 2150 MHz; lab port, no other traffic, 3 × 1024 tokens, thinking off, T=0):
| BF16 dense | MXFP8 | NVFP4 W4A16 | |
|---|---|---|---|
| prose c1 | 26.8 | 30.5 | 34.1 t/s |
| code c1 | 33.0 | 37.1 | 41.0 t/s |
| JSON c1 | 38.7 | 45.2 | 47.7 t/s |
| C4 agg. | 60.9 | 64.8 | 69.3 t/s |
| prefill 54K | 1,720 | 1,816 | 1,656 t/s |
| tool-eval-bench (69 sc.) | — | 94 | 95 |
Needle retrieval OK at 225K and 451K. KV pool 1.71M tokens (3.26× at 512K) with nvfp4_ds_mla and a 6 GiB pin. Kernel 7.0 vs 6.17 alone was worth +2-9% on the same config.
Scheduling tip. With the default max_parallel_prefills=1, a short request waits behind a 150K prefill before it even starts (1,000 tokens of prose: 359 s). --prefill-compute-share auto --max-parallel-prefills 2 + a 1024-token prefill cap brings that to 100-160 s, with only ~12% lower aggregate prefill.
Mods:
glm53-nvidia-online-mxfp8— online MXFP8/NVFP4 of the BF16 dense layersglm53-nvidia-mtp-unquantized— the MTP layer is BF16 but undeclared; without this the drafter fails at TP2glm53-thinking-template-nvidia—enable_thinking:falseno longer leaks the reasoning intocontentfix-spec-accepted-clamp-upstream— the acceptance assert can’t kill the engine under guided decoding + spec decodeglm53-port-fork862-prefill-decode-latency— port of local-inference-lab/vllm#862
Didn’t help on this stack: MTP k=4, GLM53 L2 prefetch (budgets sized for a 128 MB L2), split-target block size with DCP2, Marlin MoE (+4-7% decode / −6% prefill), target vocab argmax, draft truncation at k=3.
What GLM-5.3-Flash stack is the top pick now? Haven’t been following for a week and it seems like you guys made huge progress.
Is euger vllm docker recipe ok now?
jayleaton tensorfold stack?
techguardcoders tensorfold stack?
mialab tensorfold stack?
Thanks!
I am using now in parallel gilesco’s stack (TensorFold 0.6.4) with applied Mia’s latest PR’s, and technigmaai’s latest stack (vLLM) - both has best overal performance and stability. You could find small perf and quality comparison of them here GLM-5.3-Flash: 320B total parameters / 18B active - #685 by ajvazan