Got QuantTrio/GLM-5.2-Int4-Int8Mix serving on an 8-node GB10 / DGX Spark cluster.
This builds on CosmicRaisins’ DCP1 work from the GLM-5.2 on a 4× GB10 cluster: ~22 tok/s decode, 256K ctx, Recipe — same family of patches, but pushed onto a newer v16 unified branch and scaled to TP8.
The image is based on a newer branch from three upstreams:
local-inference-lab/vllm @ codex/fathomless-firmament-v16-unified-20260712 (5dffea8), lukealonso/b12x @ 97b3d64, and the DCP1 patches from GitHub - CosmicRaisins/glm-5.2-gb10: GLM-5.2 (744B/40B MoE) on a 4× DGX Spark / GB10 (sm_121) cluster: portable Triton sparse-MLA kernels, a data-free expert prune, MTP draft, and a one-script bootstrap. · GitHub
The v16 branch is what unlocks the prefill jump.
On TP8 + DCP=1 the previous image capped around ~1,000 t/s prefill; on v16 it climbs to ~1,200.
Recipe + patches + Dockerfile: GitHub - ciprianveg/gb10-glm-5.2: GLM-5.2-Int4-Int8 on 8x GB10 cluster · GitHub
Net, this updated image brings ~10% faster gen speed and ~20% faster prefill on an 8× cluster at DCP=1 vs the base CosmicRaisins image.
The DCP=1-specific knobs below add another ~10%+ to gen speed, for ~20% total gen uplift over the base image on 8xgb10 TP 8.
Model
QuantTrio/GLM-5.2-Int4-Int8Mix.
Results
llama-benchy, coherent corpus, tg=1500, single stream
| Depth | Prefill (t/s) | Avg decode (t/s) | Peak decode (t/s) | TTFR (ms) |
|---|---|---|---|---|
| 0 | 1,211 ± 0.9 | 34.9 ± 2.8 | 53.5 ± 3.5 | 1,693 |
| 4k | 1,117 ± 100.7 | 38.3 ± 0.5 | 58.0 ± 0.0 | 5,461 |
| 16k | 1,215 ± 23.8 | 37.7 ± 0.0 | 58.0 ± 0.0 | 14,867 |
| 32k | 1,176 ± 4.7 | 33.3 ± 2.7 | 54.5 ± 2.5 | 28,963 |
| 100k | 1,128 ± 0.9 | 34.8 ± 3.8 | 51.5 ± 1.5 | 90,448 |
| 200k | 1,019 ± 0.0 | 37.8 ± 0.0 | 50.0 ± 0.0 | 198,327 |
So single-stream avg decode runs 33–55 t/s depending on content type — ~33–39 t/s on coherent prose (holding across 0–200K context), 40–55 t/s on coding/structured.
Prefill stays north of 1,000 t/s all the way to 200K.
By workload type
| Workload | Single req | 2 concurrent reqs |
|---|---|---|
| Coherent corpus (prose) | Avg. 33–39 t/s | ~50 t/s |
| Coding / structured content | 40–55 t/s | 60–70 t/s |
| Game bench (Snake, temp=0, no thinking) | 54.16 t/s average | — |
On coding prompts the avg single-stream gen climbs to 40–55 t/s and two concurrent requests sit at 60–70 t/s.
On plain coherent prose it’s the ~33–39 t/s avg from the table above, and two concurrent land around 50 t/s.
Coding benchmark (Snake game generation, single-stream)
=== Game Benchmark (Single-Stream, temp=0, thinking=disabled) ===
Waiting for server to be ready...
Server ready after 1s
Running game benchmark (Snake game generation)...
Completion tokens: 1500
Prompt tokens: 43
Total tokens: 1543
Wall time: 27.69s
Average tok/s: 54.16
What moved the needle
- The v16 branch — single biggest prefill lever. Older branch capped ~1000 t/s; v16’s takes it to ~1,200.
- b12x @ 97b3d64 — W4A8 MoE + unified SM120 sparse MLA + PCIe DCP collectives. Decode went from ~28-49 with MTP k=4 to the 33–55 range above.
Patches (patches/v16-final/)
Production (TP8+PP1) uses 01, 03, 04, 06 only:
| Patch | Purpose |
|---|---|
01-pr72-1-draft-dcp-config-propagation |
DCP config → draft model; prevents MTP collapse under DCP>1. From CosmicRaisins’ PR #72. |
03-draft-quant-packed-mapping |
Quantized NextN draft token mapping. Without it, quantized drafts silently build unquantized and MTP acceptance collapses. From CosmicRaisins. |
04-v16-essential |
Three fixes: DeepSeekMTP SupportsPP, stale topk_indices_buffer in flashinfer SM120 sparse MLA (PR #46994), MTP embed_tokens loading under PP. |
06-b12x-stale-topk-buffer |
Same stale-buffer fix applied to b12x_mla_sparse.py (PR #46994 Fix #4). Without it _maybe_share_lm_head swaps the indexer’s buffer but the backend keeps a stale ref → garbage DSA attention and ~30% acceptance instead of ~85%. |
PP2-only (05, 07) are in the repo but experimental — TP4+PP2 hits ~1,800 t/s prefill but MTP acceptance collapses to ~8%, so decode drops to ~12 t/s. Staying on TP8+PP1 for production.
Runtime mod
fix-fsm-toolcall (PR #44993) — fixes "Failed to advance FSM" errors during tool calling + MTP. Adding this improved also tool calling bench from 88 to 91.
DCP=1 YAML tweaks vs CosmicRaisins’ recipe
Most of the DCP=1 yaml is carried over from CosmicRaisins’ DCP1 recipe unchanged. Only three knobs differ, and the first two together add ~10%+ to gen speed on top of the image’s ~10% (so ~20% total for tp 8 dcp 1):
| Tweak | Value | Why |
|---|---|---|
VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE: "1" |
env | Treats spec-extend as decode so the B12X indexer path stays consistent. |
draft_tensor_parallel_size: 1 |
in --speculative-config |
Keeps the drafter unsharded so it isn’t paying TP8 collectives on every draft step. |
NCCL_BUFFSIZE: "16777216" |
env (16 MB, was 8 MB) | Bigger NCCL buffer for gen speed at high context — the 8 MB default starts bottlenecking the allreduce on long-context decode. |
Tool evaluation (tool-eval-bench v2.0.0)
| Metric | Score |
|---|---|
| Overall quality | 91 / 100 (★★★★★ Excellent) |
| Responsiveness | 43 / 100 (median turn 3.6s) |
| Deployability | 77 / 100 (α=0.7) |
| Pass rate | 59 passed, 8 partial, 2 failed (126/138 pts) |
| Token efficiency | 0.6 pts/1K tokens (210K total) |
| Weakest category | Toolset Scale (62%) |
Stack
| Component | Version |
|---|---|
| vLLM fork | local-inference-lab/vllm @ 5dffea8 (branch codex/fathomless-firmament-v16-unified-20260712) |
| b12x | lukealonso/b12x @ 97b3d64 |
| CUDA | 13.2.0 |
| PyTorch | 2.11.0 |
| FlashInfer | Prebuilt sm_121 wheels |
| NCCL | 2.30.4 (custom aarch64) |
| transformers | ≥5.0 (--tf5 build flag) |
Caveats
-
TP4+PP2 is not viable yet, although promissing, 1800t/s prefill but MTP acceptance collapses to ~8% on the pipeline split. Sticking with TP8+PP1.
-
This is a learning/research project built on the works of others (attributed in the repo).
Links
-
Recipe + patches + Dockerfile: GitHub - ciprianveg/gb10-glm-5.2: GLM-5.2-Int4-Int8 on 8x GB10 cluster · GitHub
-
Foundational stack this builds on: GitHub - CosmicRaisins/glm-5.2-gb10: GLM-5.2 (744B/40B MoE) on a 4× DGX Spark / GB10 (sm_121) cluster: portable Triton sparse-MLA kernels, a data-free expert prune, MTP draft, and a one-script bootstrap. · GitHub
-
vLLM fork (v16 unified branch): GitHub - local-inference-lab/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs · GitHub @
codex/fathomless-firmament-v16-unified-20260712 -
b12x (W4A8 MoE / SM120 sparse MLA): GitHub - lukealonso/b12x · GitHub @
97b3d64 -
Model: QuantTrio/GLM-5.2-Int4-Int8Mix
-
Build system: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub
