So me and FABLE had a long night to some success off the shoulders of @CosmicRaisins and @Zatz hard work. We swapped models to get our KV POOL to 875k. I feel like it seems a little slower than what I was getting with the original 200k context and the 600k context version. Maybe there is some tuning but I wanted to atleast stand this up so if anyone wants to give it a shot they can getting speeds up.
We run a 4× DGX Spark cluster (GB10, 200GbE fabric) and have been following the GLM-5.2 work in the 4x GB10 cluster thread — everything below builds directly on the recipes from that thread.
Recently madeby561 published GLM-5.2-MXFP8-NVFP4-NF3-Hybrid: the unpruned model (all 256 experts) hybrid-quantized to 327GB — top-64 experts per layer in NVFP4, the rest in a custom 3-bit “NF3” format, MXFP8 elsewhere. It targets 4× RTX PRO 6000 (sm_120/amd64), but since it’s ~78GB smaller than the Int4-Int8Mix checkpoint most of us run, we wondered what the freed memory would do for KV on Sparks.
So we pointed the existing community recipes at it — b12x/eldritch stack rebuilt for sm_121a/aarch64, TP4 + DCP4 ag_rs with Zatz’s 640K launch settings as the template, MTP speculative decode. Result:
- KV pool: 877,056 tokens, currently serving at
--max-model-len 800000— clean boot, coherent output, tool calling working - Decode at 800K max-len matches a 240K-max-len boot at equal fill (shallow-context bench; we haven’t run a depth sweep yet, so no claims about decode speed at 700K fill)
Since DCP trades speed for context (as CosmicRaisins’ DCP2 results showed), the repo keeps two launch shapes:
DCP4 “context” (800K max-len, 877K pool): c1 19.9 / c2 32.5 / c3 35.3 / c4 44.1 / c6 38.7 aggregate tok/s (512-tok gens, temp 0, MTP k=4, acceptance 3.2–3.7)
DCP1 “fast” (200K max-len, 219K pool): c1 24–29 (content-dependent, acceptance 3.3–4.0) / c4 54.2 / c6 67.4 aggregate
Port notes for anyone trying this checkpoint on GB10 (it needs madeby561’s custom serving stack): the NF3 CuteDSL kernel runs on sm_121a essentially as-is — it even ships an SM121 register-count table — but three things bit us: (1) the hybrid loader hardcodes its mxfp8_tier.json path to the author’s image layout (use the HYBRID_MXFP8_TIER_JSON env override), (2) warmup needs the patched b12x_sparse_indexer_warmup.py from the author’s docker image, which isn’t in any public repo yet, and (3) eugr-style aarch64 builds need nvidia-cutlass-dsl==4.5.2 plus its .pth path hook added.
Full replication recipe, both launch envs, and all fixes: GitHub - tonyd2wild/GLM-5.2-NF3-Hybrid-4x-DGX-Spark · GitHub
Credits: madeby561 for the NF3 format, kernel, and checkpoint — this is his work, we just ported it; lukealonso for b12x and the NVFP4 donor tier; Zatz for the DCP recipe our launch config is copied from; CosmicRaisins for the stack that got GLM-5.2 running on these boxes in the first place; m9e/eugr for the build harness; aidendle94 for pointing at the checkpoint. Happy to share anything else that’s useful, and very open to tuning suggestions — single-stream on the DCP4 shape (19.9) feels like it has headroom.