Update: 512K ctx + FP8 KV + MTP-4 is now the standing config on the same 2× Spark / TP=2 setup. Repo updated.
Performance:
512K ctx · ~24–30 tok/s decode (c1, MTP-4) · 180–364 tok/s chunked prefill
KV pool 1,261,444 tokens (9 GiB fp8 pin) · 2.41× concurrency at 512K
440K-token needle retrieval byte-exact at ~52% doc depth
Progression since the day-0 post (seed-42, 88-scenario tool-call bench):
262K · bf16 KV · 311K pool → 89/100
262K · fp8 KV · 610K pool → 90/100
512K · fp8 10GiB pin · MTP-3 · 1.44M pool → 87/100
512K · fp8 9GiB pin · MTP-4 · 1.26M pool → 89/100 (current)
Two new GB10 blockers found and fixed on the way to 512K:
• --language-model-only is load-bearing here. The multimodal processor adds ~15.7 GiB to the API front-end; with gm 0.90 + pinned KV + warmup that pushes rank 0 past the 121.69 GiB UMA line → NVRM NV_ERR_NO_MEMORY ~90 s into warmup (reproduced at 262K too, so not the context length). Cost: text-only serving.
• CC-12.x sparse-MLA indexer guard: at 512K shapes the top-k wants 62 CTAs / 128 KB smem vs 48 SMs / 99 KB on GB10 → hard abort. Two-line guard (disable persistent_topk + cooperative workspace on CC family 12, falls back to the existing CUDA-safe kernel); anchors match FujitsuPolycom’s glm53-flash-tp2-spark patch.
Memory shaping that fits the remaining headroom (111.7 GiB free/rank, ~90.7 GiB weights): 9 GiB pinned KV (--kv-cache-memory-bytes, skips profiling), 4096 batched-tokens (8192 OOMs the driver at these shapes), autotune/cutedsl-warmup off in kernel-config, chunked prefill, --enforce-eager. 8 GiB pin (~1.1M pool) is the next step if you need more activation headroom.
Quality notes, since people ask why MTP at all:
• MTP is lossless w.r.t. k — every draft position is verified against the target, so k is a speed knob, not a quality knob. Per-position acceptance on this model: 0.81 / 0.67 / 0.51 / 0.42 (k=4), acceptance length ~3.1–3.4.
• The 512K profile’s −3 vs 262K+fp8 was profile-specific, not seed noise: A/B with k=4 vs k=3 recovered four of the six regressed scenarios; one (tool-output injection) fails on both 512K runs regardless of k — that ~1 point is the memory-shaping profile itself.
Extra gotchas (join the README list):
• 4 NVRM allocation-probe lines at boot are expected — don’t confuse them with the real warmup OOM (which kills VllmWorker-0 ~90 s after “GPU KV cache size” logs).
• We had image: glm53:v8 hardcoded in compose on the nodes, so an .env image change was a silent no-op — verify the running container’s image after rollouts.
• Cold boot ~14–17 min (weights ~13 + warmup); page-cache drop still in the launch ritual.
Full flag sets, the crash forensics, and the bench log are in the repo (benchmarks.md / NOTES-512k.md).