Setup: 2× DGX Spark (GB10, sm_121), CX7 200GbE direct link, TP=2. Model is the EXL3/TR3 4bpw encode (neko-legends/GLM-5.3-Flash-Uncensored-EXL3) running on the MiaAI-Lab EXL3 kit (vLLM fork) — full credit to them for the base stack. Vision tower loaded, tool calling working, OpenAI-compatible on :8888.
Honest numbers (llama-benchy + GSM8K, tables in the repo):
┌──────────────────────────────┬──────────────────────────────────────────────────────────────────────────┐
│ metric │ value │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ KV pool │ 2,196,850 tokens (fits 8 full 262K sessions) │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ concurrency slots │ 16 │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ max context │ 900K per request │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ prefill, single stream │ ~680–705 tok/s, flat from 2K to 32K prompts, no depth degradation to 32K │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ decode, single stream │ 20–24 tok/s (peak 27–29), no depth degradation │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ aggregate, steady mixed load │ 67.6 tok/s (GSM8K, 12-way) │
├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ GSM8K under 12-way load │ 59/60, zero request errors │
└──────────────────────────────┴──────────────────────────────────────────────────────────────────────────┘
What’s new: NVFP4 KV cache on the sparse-MLA path. The stock layout is 656 B/token (fp8_ds_mla); we store the latent as 288 B/token NVFP4 (packed e2m1 + per-16 e4m3 block scales). The prebuilt FlashInfer sparse-MLA kernel obviously can’t read NVFP4 — but it never has to: DSA only touches its top-k
selected rows, so a small Triton kernel gathers + dequantizes exactly those rows into a page-shaped fp8 scratch the kernel accepts (the 4×fp32 per-128-group scale words matched byte-for-byte against what concat_and_cache_mla writes). Cost: 0.36 ms/layer at decode, ≤5.6 ms/layer worst-case prefill.
Greedy outputs come out byte-identical to fp8 KV on our probes; GSM8K is at parity. ~2.3× pool density for ~3% overhead.
The part that will save you a power cycle: GB10’s unified memory livelocks instead of OOM-killing — the node stays pingable, ssh dies at banner exchange, and only the power button recovers it. The kernel stack (captured via py-spy/journalctl before one death) is the NVIDIA driver spinning in
try_to_free_pages with nothing reclaimable. Three host-side fixes turned this failure class into ordinary process restarts for us:
- zram swap (6G, prio 100) — the stock 16G file swap is too slow to absorb reclaim spikes; compressed RAM swap gives reclaim a real target
- A 0.2s-cadence memory watchdog that SIGTERMs vLLM below 0.75 GiB free (TERM first matters: raw SIGKILL mid-CUDA leaked ~20 GiB of driver memory that only returned after minutes of quiesce)
- vm.swappiness=180, vm.watermark_scale_factor=100 — and don’t go higher on the watermark: at 500 it walls ~12 GiB out of MemAvailable and looks exactly like a leak
Other traps documented in the repo’s troubleshooting table: the UMA startup free-memory check failing against page cache, MAX_MODEL_LEN reduction doubling per-token pool cost via hybrid block-id overhead, the fat-expert MoE fallback’s per-expert host syncs wedging the engine on 50–100K prompts (fix: the
kit’s GPU row-tiling, but only on a self-built image so Python and exllamav3_ext match), and a scheduler interleave policy we added so concurrent prefills don’t starve behind active decodes.
Known trade-offs, stated plainly: we run the model’s native MTP for speculation rather than DFlash2 — DFlash2 is ~5 tok/s faster on code but its ~10 GiB footprint is incompatible with the 2M+ pool on 121.63 GiB/node. And under simultaneous long-context arrivals, decoders run in waves (~50% duty, bursts
to 48–55 tok/s total) while the prefill backlog clears at ~585 tok/s — on one TP group sharing every cycle, the alternatives are prefill starvation or uniformly slow decode, so we took the waves.
Repo (MIT, everything included — patches, watchdog, dashboard, bench tables):
Credits: MiaAI-Lab (kit), turboderp (ExLlamaV3), neko-legends (encode), zai-org (model).