Got GLM-5.2 (GlmMoeDsa / DeepSeek-Sparse-Attention arch) serving on 4× GB10 over a 100G MikroTik CRS504 switch. It was quite the hassle to get DSA + MTP running, but then again Claude did 90% of the work : )
Model
cyankiwi/GLM-5.2-AWQ-INT4, with a data-free 15% routed-expert prune (256→218 experts/layer, dropping highest e_score_correction_bias — pure byte-level safetensors surgery on the INT4, no dequant). ~636B total / ~40B active. The prune frees ~12 GiB/node, which is what makes cudagraph + MTP + 256K KV all fit.
MTP gave the biggest uplift; CUDA Graph only made a 3% or so difference. With optimizations and b12x, I got prefill from 250 to ~500, but couldn’t get it any higher.
I’ve only tested a few prompts with the 15% prune in Pi, no idea on the real-world SWE performance yet so this is more of a proof of concept.
Note: This is more of a learning/research project built on the works of others (attributed in github and hf). The 15% data-free prune might not be stable. Although it has been working fine in lighter workloads.
Apparently that’s a decent reap for coding according to people in the rtx 6000 discord. You should theoretically see better prefill with the hardware acceleration.
I tried another REAP NVFP4. It ran slightly slower compared to the AWQ (~10 tok/s vs ~13 tok/s without MTP), and got itself into a loop over “What’s the capital of France?” So I didn’t continue down the REAP path. I might try this if I need longer ctx or if the 15% data-free prune I did lobotomized the model. I would love to see a gentler, 15 - 20% REAP though.
I’ll share the recipe and the patches after I confirm everything is working properly. I had to do a bunch of patches, quantize the mtp drafter to get it to work with the int4 awq, and prune the model, so I’m not exactly sure what the best way to share everything is.
What is the token gen speed oat 100k or 200k? Is the speculative model slowing it down over 100k? I can’t wait to try it. Thank you for your work on this.
I did try a 64k depth bench before I finished all the optimizations and IIRC the tg drop was not very significant. I think it went from ~21 to ~19, could just be run to run noise.
I’m not an expert on LLMs but judging from some of Claude’s CoT the model behaves a lot like DSV4 Flash when it comes to performance in longer context, probably because it also uses DSA.
Yeah. 500 felt pretty bad when I saw it but it’s okay-ish in actual usage over the past couple of days. I’m still looking at ways to increase it, though.
I think GB10s are better value than the Macs. It’s easy for Twitter influencers to pull out the 800 vs 300 bandwidth, and show qwen 3 running at 4x the TG speed, but I don’t know why people would want to buy these kinds of devices to run anything smaller than Qwen 27b. Macs become compute-bound way before the model gets large enough to be useful for me, and both metrics fall below a GB10 cluster.
But hey, at least you can run the 46th MLX 3.2 bit quant generated by some random Mac user on HF.
Oh! It should be safe to ignore NFS related stuff. It’s just how I serve models across my cluster because two of my nodes only have 1TB SSDs, while the other two have 4TB SSDs. I’m pushing a change to scrape it from the repo and update the readme.
Quick correction on framing: has_deep_gemm() is never made True on sm_121 — the indexer gate is bypassed via is_device_capability_family(120), and the fp8_fp4_* ops self-dispatch to the sm12x fallbacks. So a top-level deep_gemm shim isn’t the path, and either way it’s the indexer, not the decode lever — it won’t move tok/s.
Your 8→22 gap is 8 tok/s enforce-eager = no cudagraph; 22 is cudagraph FULL, which is only capture-safe with b12x installed (the glm52-b12x-sparse mod, pip install --no-deps b12x==0.23.0 + GLM52_B12X_MLA=1). Install b12x, set cudagraph_mode: FULL, drop --enforce-eager. Without b12x the Triton decode crashes capture, which is why you fell back to eager. Secondary: make sure your shim binds the fused fp8_fp4__mqa_topk_indices entrypoints (they return bool), not just fp8_mqa_logits — otherwise the indexer stays on the slow non-graph path.
i was surprised about month ago, when was able to convert calibrated nvfp4 checkpoint to mlx nvfp4 and actually gain performance and quality vs regular q4 on mlx.
But yeah, for agentic we need at least 2-3k prefill, otherwise it chokes on first prompt, and hard stop if you lost prefix cache for like 100-150k tokens session.
Thanks for the instructions. I used it with the unpruned model to run it on a cluster of 8 asus gb10. I get 26-27t/s using mtp no matter the context. I tried till 256k. Not bad.