GLM-5.2-Int4-Int8 on 8× GB10: ~1,200 t/s prefill, 33–54 t/s avg decode (generic - coding/structured)

Got QuantTrio/GLM-5.2-Int4-Int8Mix serving on an 8-node GB10 / DGX Spark cluster.

This builds on CosmicRaisins’ DCP1 work from the GLM-5.2 on a 4× GB10 cluster: ~22 tok/s decode, 256K ctx, Recipe — same family of patches, but pushed onto a newer v16 unified branch and scaled to TP8.

The image is based on a newer branch from three upstreams:
local-inference-lab/vllm @ codex/fathomless-firmament-v16-unified-20260712 (5dffea8), lukealonso/b12x @ 97b3d64, and the DCP1 patches from GitHub - CosmicRaisins/glm-5.2-gb10: GLM-5.2 (744B/40B MoE) on a 4× DGX Spark / GB10 (sm_121) cluster: portable Triton sparse-MLA kernels, a data-free expert prune, MTP draft, and a one-script bootstrap. · GitHub

The v16 branch is what unlocks the prefill jump.
On TP8 + DCP=1 the previous image capped around ~1,000 t/s prefill; on v16 it climbs to ~1,200.

Recipe + patches + Dockerfile: GitHub - ciprianveg/gb10-glm-5.2: GLM-5.2-Int4-Int8 on 8x GB10 cluster · GitHub

Net, this updated image brings ~10% faster gen speed and ~20% faster prefill on an 8× cluster at DCP=1 vs the base CosmicRaisins image.
The DCP=1-specific knobs below add another ~10%+ to gen speed, for ~20% total gen uplift over the base image on 8xgb10 TP 8.

Model

QuantTrio/GLM-5.2-Int4-Int8Mix.

Results

llama-benchy, coherent corpus, tg=1500, single stream

Depth Prefill (t/s) Avg decode (t/s) Peak decode (t/s) TTFR (ms)
0 1,211 ± 0.9 34.9 ± 2.8 53.5 ± 3.5 1,693
4k 1,117 ± 100.7 38.3 ± 0.5 58.0 ± 0.0 5,461
16k 1,215 ± 23.8 37.7 ± 0.0 58.0 ± 0.0 14,867
32k 1,176 ± 4.7 33.3 ± 2.7 54.5 ± 2.5 28,963
100k 1,128 ± 0.9 34.8 ± 3.8 51.5 ± 1.5 90,448
200k 1,019 ± 0.0 37.8 ± 0.0 50.0 ± 0.0 198,327

So single-stream avg decode runs 33–55 t/s depending on content type — ~33–39 t/s on coherent prose (holding across 0–200K context), 40–55 t/s on coding/structured.
Prefill stays north of 1,000 t/s all the way to 200K.

By workload type

Workload Single req 2 concurrent reqs
Coherent corpus (prose) Avg. 33–39 t/s ~50 t/s
Coding / structured content 40–55 t/s 60–70 t/s
Game bench (Snake, temp=0, no thinking) 54.16 t/s average —

On coding prompts the avg single-stream gen climbs to 40–55 t/s and two concurrent requests sit at 60–70 t/s.
On plain coherent prose it’s the ~33–39 t/s avg from the table above, and two concurrent land around 50 t/s.

Coding benchmark (Snake game generation, single-stream)

=== Game Benchmark (Single-Stream, temp=0, thinking=disabled) ===
Waiting for server to be ready...
Server ready after 1s
Running game benchmark (Snake game generation)...
Completion tokens: 1500
Prompt tokens: 43
Total tokens: 1543
Wall time: 27.69s
Average tok/s: 54.16

What moved the needle

  • The v16 branch — single biggest prefill lever. Older branch capped ~1000 t/s; v16’s takes it to ~1,200.
  • b12x @ 97b3d64 — W4A8 MoE + unified SM120 sparse MLA + PCIe DCP collectives. Decode went from ~28-49 with MTP k=4 to the 33–55 range above.

Patches (patches/v16-final/)

Production (TP8+PP1) uses 01, 03, 04, 06 only:

Patch Purpose
01-pr72-1-draft-dcp-config-propagation DCP config → draft model; prevents MTP collapse under DCP>1. From CosmicRaisins’ PR #72.
03-draft-quant-packed-mapping Quantized NextN draft token mapping. Without it, quantized drafts silently build unquantized and MTP acceptance collapses. From CosmicRaisins.
04-v16-essential Three fixes: DeepSeekMTP SupportsPP, stale topk_indices_buffer in flashinfer SM120 sparse MLA (PR #46994), MTP embed_tokens loading under PP.
06-b12x-stale-topk-buffer Same stale-buffer fix applied to b12x_mla_sparse.py (PR #46994 Fix #4). Without it _maybe_share_lm_head swaps the indexer’s buffer but the backend keeps a stale ref → garbage DSA attention and ~30% acceptance instead of ~85%.

PP2-only (05, 07) are in the repo but experimental — TP4+PP2 hits ~1,800 t/s prefill but MTP acceptance collapses to ~8%, so decode drops to ~12 t/s. Staying on TP8+PP1 for production.

Runtime mod

fix-fsm-toolcall (PR #44993) — fixes "Failed to advance FSM" errors during tool calling + MTP. Adding this improved also tool calling bench from 88 to 91.

DCP=1 YAML tweaks vs CosmicRaisins’ recipe

Most of the DCP=1 yaml is carried over from CosmicRaisins’ DCP1 recipe unchanged. Only three knobs differ, and the first two together add ~10%+ to gen speed on top of the image’s ~10% (so ~20% total for tp 8 dcp 1):

Tweak Value Why
VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE: "1" env Treats spec-extend as decode so the B12X indexer path stays consistent.
draft_tensor_parallel_size: 1 in --speculative-config Keeps the drafter unsharded so it isn’t paying TP8 collectives on every draft step.
NCCL_BUFFSIZE: "16777216" env (16 MB, was 8 MB) Bigger NCCL buffer for gen speed at high context — the 8 MB default starts bottlenecking the allreduce on long-context decode.

Tool evaluation (tool-eval-bench v2.0.0)

Metric Score
Overall quality 91 / 100 (★★★★★ Excellent)
Responsiveness 43 / 100 (median turn 3.6s)
Deployability 77 / 100 (α=0.7)
Pass rate 59 passed, 8 partial, 2 failed (126/138 pts)
Token efficiency 0.6 pts/1K tokens (210K total)
Weakest category Toolset Scale (62%)

Stack

Component Version
vLLM fork local-inference-lab/vllm @ 5dffea8 (branch codex/fathomless-firmament-v16-unified-20260712)
b12x lukealonso/b12x @ 97b3d64
CUDA 13.2.0
PyTorch 2.11.0
FlashInfer Prebuilt sm_121 wheels
NCCL 2.30.4 (custom aarch64)
transformers ≥5.0 (--tf5 build flag)

Caveats

  • TP4+PP2 is not viable yet, although promissing, 1800t/s prefill but MTP acceptance collapses to ~8% on the pipeline split. Sticking with TP8+PP1.

  • This is a learning/research project built on the works of others (attributed in the repo).

Links

Since you are using the b12x kernel, could you please try and run this on 5 nodes? Would be very interesting to see what the speed difference is.

Do you have any DCP4 benchmark results or performance data that we can refer to?

no, sorry, DCP 4 on tp8? I didn’t see the need for it.

Awesome! If I win the lottery I’ll get 4 more.

There will be signs.. :)

Due to the need for more concurrency, I used DCP4 on an eight-node cluster and achieved a more than 3.2M KV tokens size, can go with 320K ctx × 10. However, the performance was not as good as expected, and with DCP4, the prefill process significantly impacted decode performance.

I adjusted the prefill scheduler, which improved the situation, but the results were still not ideal. Thanks for your post—I’ll continue looking into possible ways to improve it.

Below is the c1 performance results I measured, tested by llama-benchy:

I see dcp taking a hit. Dcp 1 would get you 30% faster prefill and 60% faster gen speed from my tests. But if context size is important, dc4 results are not bad.

What did you adjust in prefill scheduler, relevant also for dcp 1?

This issue is not very noticeable with DCP1. It is likely caused by distributed KV in DCP4, where concurrent prefill requests starve ongoing decode requests.
From what I observed with DCP4, during the prefill phase, the decode throughput typically drops to only 0.0–0.2 tok/s and remains there until the prefill completes.

This change only mitigates the issue by allowing decode to make occasional progress, so you may intermittently see tokens being generated. However, it does not completely resolve the starvation problem.

Below are the report and patch generated by Codex 5.6:

GLM-5.2 Decode-Aware Custom Scheduler Change Guide

Updated: 2026-07-15

Purpose

This change prevents multiple long-prefill requests from blocking an existing
decode stream for extended periods.

It addresses scheduler fairness only. It does not increase KV-cache capacity,
maximum context length, or concurrency capacity.

Why It Was Changed

An earlier experiment only reduced LONG_PREFILL_TOKEN_THRESHOLD:

Threshold Maximum decode stall
2048 approximately 14.0s
1024 approximately 7.2s
512 approximately 3.6s
256 approximately 1.91s

A smaller threshold reduced decode stalls, but it also fragmented prefill when
no decode request was active. A static threshold of 256 made prompt ingestion
too slow for production.

What Changed

The custom scheduler applies dynamic limits:

  • Without active decode, aggregate prefill may use up to 16384 tokens.
  • With active decode, all prefill requests share a 1024-token budget.
  • At most one long-prefill request is selected per step.
  • Long-prefill requests rotate using round-robin/aging.
  • The 16384-token idle budget returns immediately after decode ends.
  • The feature can be disabled without removing the patch.

New settings:

ENABLE_DECODE_AWARE_PREFILL=1
DECODE_PREFILL_TOKEN_BUDGET=1024
IDLE_PREFILL_TOKEN_BUDGET=16384
MAX_LONG_PREFILLS_PER_STEP=1

Current related settings:

MAX_NUM_BATCHED_TOKENS=16384
LONG_PREFILL_TOKEN_THRESHOLD=2048
DCP4 + MTP3
320K maximum context
10 sequences
46GB/rank KV cache

Patch Scope

Validated baseline:

repository: https://github.com/local-inference-lab/vllm
commit: a663653d8cf3a66ee3c0060aea8c2fd28e3f1362

The patch changes five vLLM files to add scheduler configuration, CLI
parameters, dynamic scheduling logic, and tests. It does not include DSA
indexer, b12x, MTP mapping, NCCL, or model changes.

Test Results

Automated validation:

3 new scheduler behavior tests: PASS
7 existing prefill/throttle regressions: PASS
ruff: PASS
4 CLI parsing checks: PASS

On-hardware test using 8× DGX Spark GB10:

Workload: 1 continuous decode + 4 approximately 8K prefills

idle prefill throughput:       831.6 tok/s
pressure prefill throughput:   735.8 tok/s
pressure decode throughput:      2.74 tok/s
maximum decode stall:             1.64s
decode after prefill:             26.0 tok/s
prefill completion:               4/4
capacity-waiting maximum:         0

Conclusion

The change achieved:

  • Multiple prefills no longer stop decode for an extended period.
  • Idle prefill retains substantially higher throughput.
  • All four prefill requests complete without starvation.

Remaining limitations:

  • Decode falls to approximately 2.74 tok/s under prefill pressure.
  • Streaming can still pause for approximately 1.64 seconds.
  • The patch reduces starvation but does not eliminate all prefill impact on
    decode.

The current production configuration keeps this scheduler as the DCP4,
320K×10 baseline.

Rollback

Set ENABLE_DECODE_AWARE_PREFILL to 0 and restart the vLLM service to restore
the original scheduler path. The patch can remain installed.

glm52-decode-aware-custom-scheduler-patch-20260715.tar.gz (12.6 KB)

Very useful also for dcp 1 at long prefil ingestion and parallel requests.

I added it to the repo. Thank you!

Have you looked at what gpu utilisation is like on tp=8? I’m wondering about how much of the speed is network bandwidth limited. Wondering whether tp=4,pp=2 offers a bump in that case?

Tp 4 pp2 brings prefill from 1200 to 1800 tps, but mtp has low acceptance in pp 2, still fighting with this issue before adding pp 2 solution

I updated the image based on v18 and I went from 1200 prefill to 1300 tps, cca 10% improvements, I will publish the new docker image tomorrow.:
v18 stack: local-inference-lab/vllm @gilded-gnosis-v18gilded-gnosis-v18 (264bce1d) + b12x bc85ef3
Built from blac@7f3cbc6well-llm-docker @7f3cbc6, adapted for aarch64/SM121.

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
GLM-5.2-Int4-Int8 pp2048 1329.72 ± 0.00 1542.44 ± 0.00 1540.17 ± 0.00 1542.44 ± 0.00
GLM-5.2-Int4-Int8 tg1500 35.21 ± 0.00 57.00 ± 0.00
GLM-5.2-Int4-Int8 pp2048 @ d16000 1319.37 ± 0.00 13681.53 ± 0.00 13679.26 ± 0.00 13683.24 ± 0.00
GLM-5.2-Int4-Int8 tg1500 @ d16000 41.80 ± 0.00 66.00 ± 0.00
GLM-5.2-Int4-Int8 pp2048 @ d100000 1202.08 ± 0.00 84895.04 ± 0.00 84892.77 ± 0.00 84901.05 ± 0.00
GLM-5.2-Int4-Int8 tg1500 @ d100000 34.99 ± 0.00 45.00 ± 0.00

v18 build is live. Full update on what changed and the final numbers.

What changed (v16 → v18)

Component v16 v18
vLLM fathomless-firmament @ 5dffea8 gilded-gnosis-v18 @ 264bce1d
B12X lukealonso/b12x @ 97b3d64 voipmonitor/b12x @ bc85ef3
FlashInfer Prebuilt sm_121 wheels voipmonitor/flashinfer @ 801d57a
DeepGEMM — a6b593d
InstantTensor — 85e7c5f
PyTorch 2.11.0 2.12.0
CUDA 13.2.0 13.2.1
New PRs — #109, #111, #113, #115–#118, upstream #47979

Final benchmarks (llama-benchy, coherent corpus, MTP k=4, TP8/DCP1)

Test t/s Peak t/s TTFR (ms)
pp2048 1,329.72 — 1,542
tg1500 35.21 66.00 —
pp2048 @ d16000 1,319.37 — 13,682
tg1500 @ d16000 41.80 66.00 —
pp2048 @ d100000 1,202.08 — 84,895
tg1500 @ d100000 34.99 45.00 —
  • Prefill: +10% (1,200 → 1,330 t/s)
  • Peak decode: 66 t/s (+23% vs v16’s 54)
  • Avg decode: 35 t/s at 0 ctx, 42 t/s at 16K ctx, 35 t/s at 100K ctx

Important: First boot JIT-compiles CuTe DSL + Triton kernels. Expect latency spikes during the first ~20 requests. Mount the cache directories on the host (documented in the repo README) for persistence across restarts.

Repo restructured — both versions cleanly separated:

  • v18/ — current production (Dockerfile, build script, 7 runtime mods, recipe)
  • v16/ — legacy fallback

Full build docs, recipe, and all mods:
github.com/ciprianveg/gb10-glm-5.2

What’s different between the two b12x versions?

Prebuilt v18 Docker images are now on GHCR — no build required.

Two variants:

Tag What’s inside
v18-base Compiled stack (vLLM, B12X, FlashInfer, DeepGEMM, NCCL 2.30.4) + source patches. No runtime mods.
v18-prod Base + all 7 production runtime mods baked in (decode-aware scheduler, FSM tool-call fix, CuTe cache key fix, DSA block-table fix, MTP quant mapping, stale topk fix, venv path symlink). Zero-config deploy.
# Zero-config production image
docker pull ghcr.io/ciprianveg/gb10-glm-5.2:v18-prod

@ciprianveg could you please share what is your topo of the 8 nodes to run the GLM 5.2? Are you using any switch to talk to the 8 nodes? If you are, can you please share the which model swtich you are using?

These are my test results with DCP4. I traded some throughput to achieve more context window.
KV capacity: 3,362,048 tokens (approximately 10.51 × 320K)

V18:

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
glm-5.2 pp2048 804.13 ± 95.66 2472.83 ± 401.53 2261.63 ± 401.53 2472.83 ± 401.53
glm-5.2 tg128 27.82 ± 3.46 36.00 ± 2.16
glm-5.2 pp2048 @ d4096 846.03 ± 36.98 6659.30 ± 307.01 6448.10 ± 307.01 6659.30 ± 307.01
glm-5.2 tg128 @ d4096 25.89 ± 0.29 37.33 ± 2.62
glm-5.2 pp2048 @ d8192 865.73 ± 7.13 10550.25 ± 207.25 10339.05 ± 207.25 10550.25 ± 207.25
glm-5.2 tg128 @ d8192 27.35 ± 1.53 34.67 ± 0.94
glm-5.2 pp2048 @ d16384 853.41 ± 2.72 19537.24 ± 190.75 19326.04 ± 190.75 19537.24 ± 190.75
glm-5.2 tg128 @ d16384 27.77 ± 1.99 37.33 ± 1.70
glm-5.2 pp2048 @ d32768 851.04 ± 2.96 36851.41 ± 151.14 36640.21 ± 151.14 36851.41 ± 151.14
glm-5.2 tg128 @ d32768 31.39 ± 2.24 39.67 ± 0.47

I use a MikroTik CRS804 switch to connect eight GB10 systems.

MIKROTIK CRS804-4DDQ