# GLM-5.3-Flash on 2× DGX Spark — NVFP4 KV cache (288 B/token), 2.2M-token pool, 16-way serving, 900K context: full recipe **URL:** \ **Category:** DGX Spark / GB10\ **Tags:** llama\ **Created:** [September 2, 2026, 8:54pm UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120 "2026-09-02T20:54:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Sparkdown\_Format](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Sparkdown\_Format](https://forums.developer.nvidia.com/u/Sparkdown_Format)\ **Post date:** [September 2, 2026, 8:54pm UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/1 "2026-09-02T20:54:42Z")
Setup: 2× DGX Spark (GB10, sm\_121), CX7 200GbE direct link, TP=2. Model is the EXL3/TR3 4bpw encode (neko-legends/GLM-5.3-Flash-Uncensored-EXL3) running on the MiaAI-Lab EXL3 kit (vLLM fork) — full credit to them for the base stack. Vision tower loaded, tool calling working, OpenAI-compatible on :8888. Honest numbers (llama-benchy + GSM8K, tables in the repo): ```auto ┌──────────────────────────────┬──────────────────────────────────────────────────────────────────────────┐ │ metric │ value │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ KV pool │ 2,196,850 tokens (fits 8 full 262K sessions) │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ concurrency slots │ 16 │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ max context │ 900K per request │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ prefill, single stream │ ~680–705 tok/s, flat from 2K to 32K prompts, no depth degradation to 32K │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ decode, single stream │ 20–24 tok/s (peak 27–29), no depth degradation │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ aggregate, steady mixed load │ 67.6 tok/s (GSM8K, 12-way) │ ├──────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤ │ GSM8K under 12-way load │ 59/60, zero request errors │ └──────────────────────────────┴──────────────────────────────────────────────────────────────────────────┘ ``` What’s new: NVFP4 KV cache on the sparse-MLA path. The stock layout is 656 B/token (fp8\_ds\_mla); we store the latent as 288 B/token NVFP4 (packed e2m1 + per-16 e4m3 block scales). The prebuilt FlashInfer sparse-MLA kernel obviously can’t read NVFP4 — but it never has to: DSA only touches its top-k selected rows, so a small Triton kernel gathers + dequantizes exactly those rows into a page-shaped fp8 scratch the kernel accepts (the 4×fp32 per-128-group scale words matched byte-for-byte against what concat\_and\_cache\_mla writes). Cost: 0.36 ms/layer at decode, ≤5.6 ms/layer worst-case prefill. Greedy outputs come out byte-identical to fp8 KV on our probes; GSM8K is at parity. ~2.3× pool density for ~3% overhead. The part that will save you a power cycle: GB10’s unified memory livelocks instead of OOM-killing — the node stays pingable, ssh dies at banner exchange, and only the power button recovers it. The kernel stack (captured via py-spy/journalctl before one death) is the NVIDIA driver spinning in try\_to\_free\_pages with nothing reclaimable. Three host-side fixes turned this failure class into ordinary process restarts for us: 1. zram swap (6G, prio 100) — the stock 16G file swap is too slow to absorb reclaim spikes; compressed RAM swap gives reclaim a real target 2. A 0.2s-cadence memory watchdog that SIGTERMs vLLM below 0.75 GiB free (TERM first matters: raw SIGKILL mid-CUDA leaked ~20 GiB of driver memory that only returned after minutes of quiesce) 3. vm.swappiness=180, vm.watermark\_scale\_factor=100 — and don’t go higher on the watermark: at 500 it walls ~12 GiB out of MemAvailable and looks exactly like a leak Other traps documented in the repo’s troubleshooting table: the UMA startup free-memory check failing against page cache, MAX\_MODEL\_LEN reduction doubling per-token pool cost via hybrid block-id overhead, the fat-expert MoE fallback’s per-expert host syncs wedging the engine on 50–100K prompts (fix: the kit’s GPU row-tiling, but only on a self-built image so Python and exllamav3\_ext match), and a scheduler interleave policy we added so concurrent prefills don’t starve behind active decodes. Known trade-offs, stated plainly: we run the model’s native MTP for speculation rather than DFlash2 — DFlash2 is ~5 tok/s faster on code but its ~10 GiB footprint is incompatible with the 2M+ pool on 121.63 GiB/node. And under simultaneous long-context arrivals, decoders run in waves (~50% duty, bursts to 48–55 tok/s total) while the prefill backlog clears at ~585 tok/s — on one TP group sharing every cycle, the alternatives are prefill starvation or uniformly slow decode, so we took the waves. Repo (MIT, everything included — patches, watchdog, dashboard, bench tables): > **[GitHub - Plaaasma/glm53-flash-dual-dgx-spark](https://github.com/Plaaasma/glm53-flash-dual-dgx-spark)** > > Contribute to Plaaasma/glm53-flash-dual-dgx-spark development by creating an account on GitHub. Credits: MiaAI-Lab (kit), turboderp (ExLlamaV3), neko-legends (encode), zai-org (model). ---
**Author:** ![Sparkdown\_Format](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Sparkdown\_Format](https://forums.developer.nvidia.com/u/Sparkdown_Format)\ **Post date:** [September 2, 2026, 9:57pm UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/2 "2026-09-02T21:57:07Z")
Fixed the token generation happening in waves with multiple requests, updated the repo ---
**Author:** ![Sparkdown\_Format](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Sparkdown\_Format](https://forums.developer.nvidia.com/u/Sparkdown_Format)\ **Post date:** [September 4, 2026, 3:11am UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/3 "2026-09-04T03:11:11Z")
Updated repo with some fixes Finding 1 — GPU util% lies on stall-bound steps. 96% utilization at 36W draw. Normal decode compute pulls 70W+ on these boards. If util is high and power is low, your SMs are parked on syncs, not working. Power draw is the honest signal. Finding 2 — the real culprit: ~87 host round-trips per mixed step. py-spy from the host against the worker procs (the container lacks ptrace; sudo py-spy dump --nonblocking from outside works) showed every mixed decode+prefill step paying: - 45 MLA layers taking a torch.unique gather path (my NVFP4 code — dedup needs the row count on the host, so it syncs per layer) - 42 MoE layers hitting counts.max().item() because step tokens exceeded the EXL3 fused-MoE temp-rows cap (128) Fix: extend the fixed-shape, no-dedup gather to the mixed-step ceiling (prefill chunk + max\_seqs × spec slots = 192 tokens; duplicates cost ~2ms of bandwidth, the syncs cost ~700ms), and raise EXL3\_TEMP\_ROWS\_FUSED to 192 (+8MB at MoE concurrency 6). Results, 5 streams decoding against a 40K cold prefill: ┌───────────────────────┬──────────────┬────────────┐ │ metric │ before │ after │ ├───────────────────────┼──────────────┼────────────┤ │ mixed step time │ 830ms │ ~480ms │ ├───────────────────────┼──────────────┼────────────┤ │ aggregate decode │ ~10-13 tok/s │ ~24 tok/s │ ├───────────────────────┼──────────────┼────────────┤ │ concurrent prefill │ ~150 tok/s │ ~380 tok/s │ ├───────────────────────┼──────────────┼────────────┤ │ worst inter-token gap │ 1264ms │ ~570ms │ └───────────────────────┴──────────────┴────────────┘ Finding 3 — three measurement traps that manufactured a fake mystery: 1. iteration\_tokens\_total flushes at request completion (at least in this build). Per-step deltas showed my prefill getting zero tokens for 70 seconds, then 23K “at once” — I burned hours on a starvation theory before a scheduler-side probe proved chunks flowed steadily the whole time. 2. Counting SSE chunks undercounts tok/s by ~τ under spec decode. Each stream chunk carries 2-3 accepted tokens. Chunk cadence measures step time (still useful!); token rate must come from engine counters or usage. 3. Stale log tails: our launcher appends across boots, so failure-signature greps kept “detecting” the previous boot’s crash. Finding 4 — MTP depth on this stack: 3, not 5. MTP-5 works on the NVFP4 stack posted here earlier, but if you run custom KV with CUDA-graph-captured gathers, watch your capture sizes: depth 5 × 16 seqs = 96-token steps, which fell off my 64-token graph-safe path and killed capture (cudaErrorStreamCaptureInvalidated). Depth 3 × 16 = 64 exactly. Measured τ 2.87 vs 2.50 at depth 2, per-position acceptance 88.5/61.2/36.8% — position 3+ would free-ride, consistent with the acceptance data in tonyd2wild’s repo. ---
**Author:** ![Sparkdown\_Format](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Sparkdown\_Format](https://forums.developer.nvidia.com/u/Sparkdown_Format)\ **Post date:** [September 4, 2026, 3:46pm UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/4 "2026-09-04T15:46:14Z")
Much better multi-agent speeds and response times, updated repo ```auto ┌───────────────────────────────────┬──────────────────────────┬─────────────────────────┐ │ scenario │ session start │ now │ ├───────────────────────────────────┼──────────────────────────┼─────────────────────────┤ │ 78K prompt re-submitted exactly │ 22.7s (replay ~15K) │ 0.3s (replay 7 tokens) │ ├───────────────────────────────────┼──────────────────────────┼─────────────────────────┤ │ 78K + 2K new (agent-turn shape) │ 12.8s (replay ~8K) │ 2.9s (replay 1.3K) │ ├───────────────────────────────────┼──────────────────────────┼─────────────────────────┤ │ c8 short-context aggregate │ 53.4 tok/s, 385ms steps │ 68.4 tok/s, 205ms steps │ ├───────────────────────────────────┼──────────────────────────┼─────────────────────────┤ │ c5 decode during 40K cold prefill │ ~10–13 tok/s (yesterday) │ 20–23 tok/s │ └───────────────────────────────────┴──────────────────────────┴─────────────────────────┘ ``` ---
**Author:** ![Sparkdown\_Format](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Sparkdown\_Format](https://forums.developer.nvidia.com/u/Sparkdown_Format)\ **Post date:** [September 11, 2026, 3:09pm UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/5 "2026-09-11T15:09:10Z")
Have done a few more stability updates to the repo for multi agent workflows feeling a bit smoother. Also improved prefill quite a bit and am peaking at ~1600 tok/s prefill instead of the ~900 I was getting before. I’ve been using this daily for a few days in deepseek harness coding various projects and it really does feel like using opus 4.8, it’s so amazing we can run this in our homes. ---
**Author:** ![ammarabbaxi13](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@ammarabbaxi13](https://forums.developer.nvidia.com/u/ammarabbaxi13)\ **Post date:** [September 23, 2026, 5:45am UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/6 "2026-09-23T05:45:01Z")
whats the token gen speed? ---
**Author:** ![Sparkdown\_Format](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Sparkdown\_Format](https://forums.developer.nvidia.com/u/Sparkdown_Format)\ **Post date:** [September 23, 2026, 4:36pm UTC](https://forums.developer.nvidia.com/t/glm-5-3-flash-on-2x-dgx-spark-nvfp4-kv-cache-288-b-token-2-2m-token-pool-16-way-serving-900k-context-full-recipe/382120/7 "2026-09-23T16:36:34Z")
| **decoding streams** | **samples** | **aggregate tok/s** | **per stream** | **draft acceptance** | **accepted tokens/step** | | --- | --- | --- | --- | --- | --- | | 1 | 75,312 | 22.7 | 22.7 | 58% | 2.75 | | 2 | 12,306 | 35.5 | 17.7 | 58% | 2.71 | | 3 | 3,421 | 43.4 | 14.5 | 70% | 2.41 | | 4 | 2,321 | 41.2 | 10.3 | 71% | 2.42 | | 5 | 231 | 48.7 | 9.7 | 67% | 2.33 | | 6 | 54 | 55.7 | 9.3 | 65% | 2.30 |