GLM-5.2 IQ4_XS on 4× GB10 — 6.28 tok/s, DSA active, full recipe
4.6 was so proud we decided to share our first GLM-5.2 run on a four-node DGX Spark cluster.
Everything you need to reproduce it is below.
HARDWARE
- 4× MSI EdgeXpert GB10, 128GB unified memory each
- MikroTik CRS812 switch, 200G QSFP56 DAC (one cable per node)
- DGX OS 7.5.0, Driver 580.142, CUDA 13.0
- Firmware: EC 10700, SoC 10800 (see note at bottom — this matters)
MODEL
- unsloth/GLM-5.2-GGUF, UD-IQ4_XS (~365GB across 4 nodes)
- MIT license, 744B total / 40B active, 256 experts (8 per token)
- 1M context window
BUILD
git clone GitHub - ggml-org/llama.cpp: LLM inference in C/C++ · GitHub
cd llama.cpp
cmake -B build
-DGGML_CUDA=ON
-DCMAKE_CUDA_ARCHITECTURES=121a-real
-DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
LAUNCH (4-node RPC)
Nodes 2-4 (RPC workers):
./build/bin/llama-rpc-server --host 0.0.0.0 --port 50052
Node 1 (head — serves the model):
./build/bin/llama-server
–model <path-to-iq4_xs-shards>/GLM-5.2-IQ4_XS-00001-of-00009.gguf
–rpc 10.10.10.2:50052,10.10.10.3:50052,10.10.10.4:50052
–ctx-size 1048576
–n-gpu-layers 999
–port 8080
–host 0.0.0.0
Transport is RPC over TCP/IP across the QSFP fabric. Not RDMA —
llama.cpp RPC doesn’t use NCCL.
RESULTS
All numbers measured, not estimated.
Smoke tests: 3/3
- Arithmetic (2+2) ✓
- Code gen (Python function) ✓
- 1M-context needle retrieval ✓
Finance quality (ES futures margin calc, temp 0, max reasoning): 12/12
- Model inferred the $50/point multiplier without being told
- Correctly handled the “at maintenance, not below” edge case
Throughput:
C=1: 6.28 tok/s decode (rock stable, 6.27-6.29)
C=4: 9.07 aggregate / 2.32 per-request
C=8: 8.50 aggregate — drops below C=4
Concurrency doesn’t scale. RPC pipeline serialization is the ceiling.
This is a single-user research model on llama.cpp, not a throughput server.
Load time: ~24 min (CPU-bound IQ4_XS repack on ARM, one-time cost).
DSA STATUS: ACTIVE
Confirmed through the runtime path, not inferred from the model name.
glm-dsa.cpp selected, lightning indexer executing, sparse KV cache active.
This is real DSA, not a dense-MLA fallback.
MTP STATUS: NOT ACTIVE
The GGUF carries the NextN/MTP tensors (block 78). llama.cpp supports
–spec-type draft-mtp. But glm-dsa.cpp explicitly skips the NextN layers
when building the graph. MTP is in the weights but not wired to the
glm-dsa path.
VLLM / SGLANG ON GB10: BLOCKED
DSA through vLLM or SGLang hits the DeepGEMM arch-gate on SM121
(arch_major must be 9 or 10; SM121 = 12). Same wall as GLM-5.1.
vLLM PR #43477 targets SM120 — unclear if it extends to SM121 and GLM-5.2.
Our separate b12x kernel verification (June 16) confirmed sparse-MLA runs
correct on SM121 (cos≈0.99999). The kernels work. The engine glue doesn’t.
ASKS
- MTP + glm-dsa: Anyone wiring the NextN/MTP head into the glm-dsa graph?
That’s the biggest speed lever. - vLLM/SGLang DSA on SM121: Anyone testing PR #43477 on GB10 with GLM-5.x?
- Multi-node transport: Anyone running GLM 5.x multi-node with something
faster than llama.cpp RPC?
FIRMWARE NOTE (helps everyone, not GLM-specific)
If you’re running MSI EdgeXpert and haven’t checked for firmware updates:
do it now. We had EC 10600 and SoC 10700 sitting. Flashing
to EC 10700 / SoC 10800 cleared a stuck power state that was costing us
on every model. Our 397B SGLang went from 12.8 to 20.83 tok/s after the flash.to a 41-46 C=1 range with MTP. Check: sudo fwupdmgr get-updates
CREDITS
- ht12 — firmware pointer that solved our 397B throughput gap
- lukealonso — b12x sparse-MLA verification on SM121
- Unsloth — GLM-5.2 GGUF quantization
- ggml-org — upstream llama.cpp with glm-dsa support
- LA (Claude Opus 4.6) — cluster architecture, benchmark methodology,
skill documentation, elimination ladder that traced the firmware root cause
— korpy
