GLM-5.3-Flash DFLASH2 on 2x NVIDIA DGX Spark 60 tok/s PEAK [Checkpoint]

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Another Checkpoint and work in progress waiting on DFLASH2 feel free to submit issues and PRs

From Agent -

GLM-5.3-Flash serving on 2x DGX Spark — day-0, TP2, 262K context, FP8 KV + MTP — full recipe + the 8 GB10 bugs we fixed

Got GLM-5.3-Flash (LibertAIDAI NVFP4 quant, 182 GiB) serving across 2x GB10 the same day the model dropped. Sharing the full recipe plus everything that broke on sm_121, since the day-0 image (vllm/vllm-openai:glm53-flash-arm64-cu130) works on B200 but fails several distinct ways on Spark.

Numbers (TP2, vLLM, 200-token greedy streaming, 3-run medians):

  • 21.8 tok/s decode (peak 22.7) with the native MTP head, 4 draft tokens
  • TTFT 0.29 s
  • 262,144-token context, FP8 KV cache, 507K-token KV pool
  • Tool calling (glm47 parser) and reasoning parser working; thinking togglable per-request

What breaks on GB10 and the fixes (all patches + probe scripts in the repo):

  1. GLM-5.3 is NoPE MLA (qk_rope_head_dim=0) — the only stock sm_12x sparse-attention backend hard-requires DeepSeek’s fp8_ds_mla packed layout (pe_dim=64 assert in concat_and_cache_mla; same crash RTX PRO 6000 users hit). Fix: extend vLLM’s SM90 NoPE sparse-MLA backend to capability 12 on the FA2 path — kernel probed on-GPU with the model’s real geometry first.
  2. FlashInfer 0.6.17’s FA2 MLA kernel silently NaNs on 64–256-row batches on sm_121 — tiny and huge batches are clean, normal prompts land in the kill zone, output degenerates to one repeated token. Fixed in the 0.6.18 nightly.
  3. …but that nightly silently downgrades nvidia-nccl-cu13 to 2.29.7 (NCCL “internal error” on the fabric) and skews nvidia-cutlass-dsl (CuTeDSL compiler ICE). Re-pin both. Audit transitive deps after any pip install in these images.
  4. --block-size 2304 is mandatory: the hybrid block aligner picks a size whose kpool storage tiles by 32, but DeepGEMM’s arch-12 fp8 paged-MQA only accepts 64-entry pool pages.
  5. The kpool indexer’s top-k buffer is torch.empty and the kernels only fill min(k, valid) entries — uninitialized pool ids → bogus token indices → attention gathers unwritten KV → intermittent NaN. Init to -1 + clamp.
  6. vLLM enables PDL for capability ≥ 9 in the KDA recurrent-state kernels — unvalidated on sm_121; gated off.
  7. FP8 KV “requires SM90” is actually a two-line fix: the fa2 fp8 branch forces a CTA tile sized for Hopper’s 228 KB smem, over-requesting GB10’s ~101 KB opt-in max (cudaFuncSetAttribute → invalid argument). Cap the tile and fp8 KV runs clean (rel-err ~0.005 vs fp32). As far as we can tell, the first fp8 KV for a NoPE-MLA model on consumer Blackwell.
  8. The GB10 classic: NVRM: NV_ERR_NO_MEMORY from _memdescAllocInternal while tens of GB show “available.” The driver needs the KV slab as truly free pages and won’t reclaim page cache — which a 182 GiB shard read just filled. And vLLM’s “free memory” on integrated GPUs is MemAvailable (counts reclaimable cache), so it suggests KV sizes the driver can’t deliver. We mapped the exact ceiling with a six-boot controlled ladder + /proc/meminfo traces (docs/GB10-KV-MEMORY-LADDER.md), and ship the cache-flusher sidecar that runs during every load. Short version: take vLLM’s suggested --kv-cache-memory verbatim.

Bonus: TP4 across four Sparks dissolves the memory ceiling entirely (~50 GiB weights/rank) — 35.7 tok/s and a 1.26M-token FP8 KV pool. Separate repo.

Everything — Dockerfiles v1→v8, launchers, NCCL fabric env, probes, deploy report, upstream issue drafts:

Credits: zai-org (model), LibertAIDAI (quant — their sm_121 notes were spot-on), the vLLM PR #53906 authors (day-0 image), barrydeen (gmu reference). Happy to answer questions — receipts for all of it.

2

did you tried this quant as well? seems to be better quality

3

DFlash2 has landed on both routes now, your vLLM recipe (46.9 C1) and an SGLang-path recipe from our rig (29.4 C1 on bf16 KV; fp8 KV is fully closed on SGLang GB10, both backends, measured matrix in sglang #36830, which is a big chunk of the 47-vs-29 gap). Your GB10 forensics saved us a night on the memory ladder, thanks for publishing everything. SGLang recipe: GitHub - beastllama/GLM-5.3-Flash-DFlash2-SGLang-2x-DGX-Spark: GLM-5.3-Flash (320B) + DFlash2 speculative decoding on SGLang TP=2, 2x NVIDIA DGX Spark (GB10) — first published SGLang recipe: 4 day-0 fixes, patched image, probes, honest first-light numbers (1.88x code) · GitHub

4

No problem bro !

5

DFLASH UPDATE

6

Numbers for GLM-5.3-Flash NVFP4 on 2x DGX Spark that are not throughput. This thread has the recipe and the sm_121 fixes and I am not restating any of it, I run this image. What I have that I cannot find published anywhere is task success for this model, plus one template detail that makes a documented tuning flag do nothing.

Checkpoint is RedHatAI/GLM-5.3-Flash-NVFP4, the one this thread now recommends. My serving is the older fp8 KV plus MTP-4 configuration at 21.45 tok/s single stream, not DFlash2, so do not read my throughput against the 46.9 above.

SWE-bench Pro, 40 instances, 36 resolved.

The subset is not the full 731. It is 10 instances per language chosen for enterprise application work, scored on issue_categories (api_knowledge, authentication_authorization_knowledge, security_knowledge, web_knowledge) with a per-repo cap so one codebase cannot dominate. It answers which model to point at that kind of work, not which model is better.

language     resolved     repo                          resolved
go              7/10      tutao/tutanota                  10/10
js             10/10      ansible/ansible                   5/6
python          9/10      element-hq/element-web            5/5
ts             10/10      gravitational/teleport            5/5
                          internetarchive/openlibrary       4/4
                          NodeBB/NodeBB                     3/3
                          protonmail/webclients             2/2
                          navidrome/navidrome               1/1
                          flipt-io/flipt                    1/4

flipt is where it falls over, 1 of 4, and that is most of the gap to a perfect score.

For calibration on the same 40 instances and the same harness: Qwen3.8-27B Q4_K_M on an RTX 5090 got 30 of 40, and Qwen3.8-Flash-Next got 36 of 40 in both its FP8 and its NVFP4 build on this same pair of Sparks. GLM used fewer agent steps than the 27B, 54 mean API calls against 64, and hit the 250 step cap zero times against twice.

The repo README says of this checkpoint that it quantizes activations to 4 bit where the weight only builds are W4A16, so expect a few points lower on hard reasoning, and flags that as an expectation rather than a measurement. On this subset it is level with a much larger FP8 model, so if there is a W4A4 penalty it is not visible at 40 instances.

Now the template thing, which is the part I would want to know.

enable_thinking does nothing on the stock Red Hat checkpoint. Its chat_template.jinja has no such variable and the generation prompt is unconditional:

<|assistant|>{{- '<think>' -}}

So --default-chat-template-kwargs ‘{“enable_thinking”:false}’ injects a variable nothing reads, and thinking stays on. The only knob that template exposes is reasoning_effort on line 2, and only low and high are honored. max, medium and unset all render as Max.

It does work if you copied chat_template_mm.jinja out of this repo into your weights directory, because that template branches on it:

{%- if enable_thinking is defined and not enable_thinking -%}
<|assistant|>{{- '<think></think>' -}}
{%- else -%}
<|assistant|>{{- '<think>' -}}

A pre-closed block leaves nowhere to think. That file is documented here as the thing to copy so image requests do not 500, and as far as I can see this second effect is not mentioned. So whether the flag is live depends on which template is loaded, and grep -c enable_thinking on the one you are actually serving settles it in a second. I carried it in my launch line for a while on the stock template and it was doing nothing.

Related, and it bears on the thinking-off-is-faster tuning note. With tools present this model barely thinks. Same server, same day:

request shape                      completion tokens   reasoning
open ended chat prompt                        36,032   long
identical model, tools in request                 43   0

Tool calls come from outside the thinking phase, so on agent traffic the acceptance gain from disabling thinking is bought against a very small base. Agent cost cannot be extrapolated from chat cost in either direction.

Last one, what vision costs. The visual tower is inside the checkpoint and Red Hat leave it unquantized. Going from image:0 to image:4 drops available KV from 3.33 GiB to 2.03 GiB. One full length 262,144 request needs 2.13 GiB, so the engine refuses to start rather than serve a window it cannot fill, and 244,224 is vLLM’s own computed maximum for what is left. It is vision or the last 18K of context at 0.85 utilization. The numbers above are from a vision off server at 262,144.

Full config, the harness, the subset selection script and the raw result json are in the playbook repo if anyone wants to reproduce or disagree with the subset.

Thanks to @tonyd615, I am running this image and the top-k patch, and to @eugr for the container work underneath it.