GLM-5.2 (unpruned) @ 200K context on 4× DGX Spark — 27 tok/s single / 52.5 tok/s @c4
Serving GLM-5.2 QuantTrio Int4-Int8Mix (unpruned, all 256 experts — no REAP) across
4× NVIDIA DGX Spark (GB10, sm_121, aarch64, 128G unified each) via vLLM native
multi-node (TP=4, mp executor, no Ray):
⁃ ✅ 200K context — exactly 200,064-token KV pool on fp8_ds_mla (DeepSeek-style MLA KV)
⁃ ✅ 27.0 tok/s single-stream (warm median) · 30.7 @c2 · 52.5 tok/s @c4 aggregate
⁃ ✅ MTP speculative decoding (k=4, lossless) + FLASHMLA_SPARSE + DSA sparse attention
⁃ ✅ Reasoning + tool-calling parsers working (glm45 / glm47)
⁃ ➕ Optional: vision bolted on via a sidecar VLM — zero GPU cost on the cluster (see below)
This is a reproduction + extension of tonyd2wild’s recipe
(GLM-5.2-QuantTrio-200K-4x-DGX-Spark,
built on eugr/spark-vllm-docker tooling).
Our numbers confirm his @c4 (52.5 vs his 53.5) on independent hardware. What we add: one
build gotcha that cost us a full rebuild to find — worth reading even if you serve a
different model with this tooling.
⸻
⚠️ THE gotcha: “preset PRs” silently drift your pinned vLLM ref
The eugr-style Dockerfile defaults to VLLM_APPLY_PRESET_PRS=“” (auto), which applies
preset PRs (47392 + 47618) by running git merge pr-NNNN on top of the pinned ref
(ab666069). The trap:
PR branches get rebased onto newer vLLM main over time. So the merge silently pulls
in a week+ of unrelated mainline drift. Your “pinned” build isn’t pinned — our
version string showed dev851+g84710906a instead of dev190+gab6660699.
For GLM-5.2 that drift broke fp8_ds_mla KV page-padding: page_size_padded came back
None, and the plain-view reshape branch in gpu/attn_utils.py crashed with:
shape ‘[N, 64, 576]’ is invalid for input of size N64656
(656 physical = 512 fp8 + 128 bf16 rope + 16 scale; 576 logical — the padding-aware path
never engaged.)
The fix: build PURE ab666069
1. Dockerfile: ARG VLLM_APPLY_PRESET_PRS="0" — no preset merges.
2. One hard patch -p1 in the build (Gemma4 share_embeddings, llm_base_proposer.py)
3. then fails to apply — make it tolerant (|| echo skip). It's Gemma-only; GLM
4. doesn't need it. All other TEMPORARY PATCH steps are soft python/sed with skip-fallbacks.
5. Verify the built version string is exactly dev190+gab6660699.
The presets exist for OTHER models (Gemma4 / DiffusionGemma / MiniMax / Qwen). GLM-5.2’s
fp8_ds_mla support is in ab666069 itself.
General takeaway: any pin-and-merge-PRs build system drifts silently as PR branches
rebase. Always verify the final version string matches your pinned ref.
⸻
Stack
Component Value
Checkpoint QuantTrio/GLM-5.2-Int4-Int8Mix — unpruned 256 experts, 405G on disk → 98.08 GiB weights/node at TP=4
vLLM built from source at pure ab666069 (see gotcha above), eugr/spark-vllm-docker tooling
KV cache fp8_ds_mla (MLA compressed), --kv-cache-memory-bytes 10950000000, GMU 0.91 → 200,064-token pool
Speculative MTP k=4, draft_tp=1, FLASHMLA_SPARSE backend — accept length 3.0–3.2 on our pair (author: 3.3–3.6; explains our 27.0 vs his 28.8 single-stream)
Attention DSA sparse indexer; cudagraph FULL (auto → FULL_AND_PIECEWISE for the indexer)
Kernels 10 CosmicRaisins sm12x Triton kernels bind-mounted into the container
Concurrency --max-num-seqs 6 — needs the indexer MTP-overhang patch (included as a mod)
Parsers --reasoning-parser glm45 --tool-call-parser glm47
Cluster vLLM native multi-node (mp executor): head = rank0 local, workers launched via ssh — no Ray
Fabric NCCL over IB: NCCL_IB_HCA=mlx5_1, iface enp1s0f1np1 — IB confirmed by throughput (52.5 @c4 is impossible over TCP fallback)
Boot ~10.5 min (weight load + cudagraph capture)
Benchmarks (256-token probes, temp 0, warm engine)
stream ours (2026-07-12) recipe author
single (warm median) 27.0 tok/s 28.8
concurrency 2 (aggregate) 30.7 37.6
concurrency 4 (aggregate) 52.5 53.5 ✓
The single-stream gap tracks MTP accept length (3.0–3.2 vs 3.3–3.6) — workload/content
dependent, not a config difference. @c4 matches the author’s.
Optional: adding vision to a text-only GLM — sidecar VLM at the proxy
GLM-5.2 is text-only. We gave it practical image support without touching the model or
spending a single GB of cluster memory, by handling vision at the proxy layer:
client (any: OpenAI- or Anthropic-format)
│ request may contain image blocks
▼
thin proxy in front of vLLM
│ detects images → sends them to the sidecar VLM → gets a detailed description
│ → REPLACES the image block with that text → forwards to GLM-5.2
▼
GLM-5.2 (4× DGX Spark) answers over the description
⁃ Sidecar VLM: Qwen3-VL-30B-A3B (30.5B total / 3.3B active MoE, native vision,
⁃ 32-lang OCR) running on an Apple Mac Studio via MLX — a machine that was idle
⁃ anyway. The GB10 cluster stays 100% dedicated to GLM.
⁃ Captioning call: non-streaming, temperature 0, generous max_tokens (~1500) so
⁃ descriptions are deterministic and detailed. If the VLM is down, the proxy degrades
⁃ gracefully (inserts a "(vision model unavailable)" note instead of failing the request).
⁃ Client-transparent: Claude-style and OpenAI-style clients get image support with
⁃ zero client-side changes — they think the text model has eyes.
Gotcha we hit: mlx-vlm’s bundled threaded server had a streaming bug (“no Stream”
errors) on this model — we run a small custom main-thread OpenAI-compatible server
around mlx-vlm instead. If you see stream errors from mlx-vlm’s server, that’s your fix.
Honest tradeoff: this is description-based vision — the LLM reasons over a caption,
not pixels. Excellent for screenshots, documents, diagrams, UI questions and OCR-ish
tasks; weaker than a native VLM for fine-grained spatial/pixel-exact queries. For a
text-only frontier model it’s a remarkably cheap 90% solution.
Operational gotchas (all hit live)
1. Preset-PR drift (above) — the big one. Verify your version string.
2. Page-cache stall during weight load on GB10 unified memory — run a drop-caches
3. loop (echo 3 > /proc/sys/vm/drop_caches on ALL nodes every 60s) for the duration of
4. the ~98G/node load, or the load wedges.
5. --max-num-seqs > 1 needs the indexer MTP-overhang patch — without it the DSA
6. indexer overruns under concurrent MTP drafting.
7. Never SIGKILL / docker rm -f CUDA processes on GB10 — unified-memory GPU
8. allocations leak unreclaimably until reboot. Stop gracefully (SIGTERM / clean stop
9. script) if you intend to relaunch without a reboot.
10. Verify IB is actually in use — with NCCL silently falling back to TCP everything
11. still works, just slower. Aggregate throughput is the honest check.
Credits
⁃ tonyd2wild — the recipe this reproduces
⁃ (GLM-5.2-QuantTrio-200K-4x-DGX-Spark).
⁃ QuantTrio — the Int4-Int8Mix checkpoint (unpruned).
⁃ CosmicRaisins — the sm12x Triton kernel set.
⁃ eugr (Eugene Rakhmatulin) — spark-vllm-docker cluster tooling.
⁃ Zhipu AI — GLM-5.2 (check the model license for your use case).
⁃ The vLLM project.
4 consumer GB10 boxes, an unpruned 744B-class MoE at 200K context, 52 tok/s aggregate —
and the one lesson we’d underline for anyone using pin-and-merge build systems: your pin
is only as pinned as the branches you merge into it.