GLM-5.3-Flash NVFP4 on 3× DGX Spark — TP=3, 512K context, 35 tok/s

[jetspark]

I’d like to share my success story with the community.

Model: LibertAIDAI/GLM-5.3-Flash-NVFP4

Repo: GitHub - jetnet/glm53-flash-nvfp4-tp3: How to run GLM-5.3-Flash-NVFP4 on 3 x Nvidia DGX Spark (TP3) · GitHub

What is running

glm-5.3-flash via Bifrost :4000 → spark01:8045, TP=3 on 01/02/03, MTP-4, context 524.288, KV-pool 1.505.849 tok (2.87×), weights 63.79 GiB/rang, GPU clock 1500 MHz.

 ┌─────────────────────┬────────────────────────┐
 │ Metric              │ Value                  │
 ├─────────────────────┼────────────────────────┤
 │ Decode              │ 35.2 tok/s (32.1–39.0) │
 ├─────────────────────┼────────────────────────┤
 │ TTFT                │ 0.26 s                 │
 ├─────────────────────┼────────────────────────┤
 │ Prefill             │ ~1 800 tok/s           │
 ├─────────────────────┼────────────────────────┤
 │ Retrive 471.813 tok │ PASS, 274.6 s          │
 ├─────────────────────┼────────────────────────┤
 │ Xid                 │ 0 / 0 / 0              │
 └─────────────────────┴────────────────────────┘

GPU cap test

 ┌──────────┬────────────┬─────────────┐
 │          │ Decode     │ Pfefill 64K │
 ├──────────┼────────────┼─────────────┤
 │ 1500 MHz │ 35.2 tok/s │ 44.5 s      │
 ├──────────┼────────────┼─────────────┤
 │ 2100 MHz │ 36.8 tok/s │ 35.3 s      │
 ├──────────┼────────────┼─────────────┤
 │          │ +4.4%      │ −26%        │
 └──────────┴────────────┴─────────────┘

Tools

tool-eval-bench --base-url http://spark01:8045 --hardmode --parallel 1 --seed 42

Note: default reasoning mode: max

│    Model:  /models/glm-5.3-flash-nvfp4
│    Score:  90 / 100                   
│    Rating: ★★★★★ Excellent        
│    Benchmark: tool-eval-bench v2.6.1.dev24+g845f15e6c
│    Engine:       vLLM 0.1.dev20051+g487ecf187
│    Max context:  524,288 tokens
│
│    ✅ 74 passed   ⚠️  11 partial   ❌ 3 failed
│    Points: 159/176
│
│    Quality:        90/100
│    Responsiveness: 31/100  (median turn: 5.2s)
│    Deployability:  72/100  (α=0.7)
│    Weakest: I Context & State (80%)
│
│    Completed in 1883.6s
│
│    📊 Token Usage:
│    Total: 552,143 tokens  │  Efficiency: 0.3 pts/1K tokens
│
│    🛡️  SAFETY WARNINGS (1):
│      ⚠ TC-43 (Omitted Required Parameter): Called web_search with an empty query — violated required parameter constraint.

Partials

Most are earned (TC-63 never searched for a match, TC-57 answered without searching, TC-62 missing corrected revenue). But TC-28, TC-35, TC-47, TC-75 share one shape: “made an extra call” / “volunteered an unrequested conversion” / “asked for details but also guessed a date”. Those are over-helpfulness penalties, not wrong answers.

Bottom line

Of 17 lost points: 2 taken incorrectly (TC-68 should be 1/2), 2 taken pedantically (TC-43), the rest earned. True score is ~91–92, and the safety-critical warning should be ignored — it is category automation, not a risk assessment.

Performance

tool-eval-bench bench --perf-only \
     --base-url http://spark01:8045/v1 --model glm-5.3-flash --backend vllm \
     --tokenizer ~/.cache/glm53-tokenizer \
     --pp 2048 --tg 512 \
     --depth 6144,63488,161792 \
     --concurrency 1,2,3 \
     --benchy-runs 3 --benchy-latency-mode generation \
     --timeout 1800 --no-live \
     --label glm53-tp3-perf-1500mhz \
     --output-dir runs/ --json-file runs/glm53-perf.json
	 

╭──────────────────────────────────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────────────────────────────────╮
│ glm-5.3-flash                                                                                                                        │
│ pp=[2048]  tg=[512]  depth=[6144, 63488, 161792]  concurrency=[1, 2, 3]  runs=3  latency=generation                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

  ✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/27 0:58:29

  llama-benchy 0.4.0
  Estimated latency: 150.9 ms

                                                          llama-benchy Results
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Test                              ┃    c    ┃         pp t/s ┃         tg t/s ┃        TTFT (ms) ┃      Total (ms) ┃          Tokens ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ pp2048 tg512 @ d6144              │   c1    │          1,591 │           31.6 │            5,309 │          21,358 │        2048+512 │
│ pp2048 tg512 @ d6144              │   c2    │          1,522 │           36.1 │            8,068 │          32,652 │        2048+512 │
│ pp2048 tg512 @ d6144              │   c3    │          1,516 │           40.5 │           10,796 │          40,651 │        2048+512 │
│ pp2048 tg512 @ d63488             │   c1    │          1,509 │           30.8 │           43,578 │          60,061 │        2048+512 │
│ pp2048 tg512 @ d63488             │   c2    │          1,488 │           15.3 │           65,824 │          97,797 │        2048+512 │
│ pp2048 tg512 @ d63488             │   c3    │          1,482 │           13.3 │           88,085 │         135,093 │        2048+512 │
│ pp2048 tg512 @ d161792            │   c1    │          1,470 │           34.2 │          111,631 │         126,439 │        2048+512 │
│ pp2048 tg512 @ d161792            │   c2    │          1,447 │            7.5 │          169,173 │         201,702 │        2048+512 │
│ pp2048 tg512 @ d161792            │   c3    │          1,444 │            6.1 │          225,916 │         282,337 │        2048+512 │
└───────────────────────────────────┴─────────┴────────────────┴────────────────┴──────────────────┴─────────────────┴─────────────────┘

Conslusion: this is the first NVFP4 quant I’m happy with :D

A couple of screenshot from the cluster whilePi is running.

And the Pelican of course:

[jetspark]

Meantime, a new version of horizontal double tetris is ready.

One prompt - one self-contained HTML (after 1h 7min).
The best double tetris created by my local models.

[florianbrede]

Thanks for testing TP=3.

The results (and the KV pool without memory pressure) are great given the fresh state, but overall the improvements over TP=2 are not scaling too well. Is that a limitation of TP=3 or is there likely room for optimization?

My current llama-benchy results with TP=2 (MTP4, ~990 KV Pool) are:

PP

d0: 2,048 computed / ~1.31 s = 1,558–1,564 PP tok/s
d32K rep 1: 11,008 computed / 7.32 s = 1,504.5 PP tok/s
d32K reps 2–3: 6,400 computed / ~4.33 s = 1,477–1,478 PP tok/s

TG128

Context	c1	c4
d32K	29.15	40.24
d64K	32.73	41.08

I’ve been considering if a 3rd GB10 would be a good investment of whether that’s an awkward state between 2 and 4.

[vedcsolution]

more kv cache on TP3, This model is the best I’ve tried locally; it deserves a dedicated kernel.

[renek]

Which quant do you use ?

[jetspark]

I asked the agent to summarize my current setup:

---

Short answer: TP=3 is not a throughput upgrade over TP=2 for this model — it’s a memory upgrade. And most of the remaining headroom is in the engine config, not in adding ranks.

Our TP=3 numbers next to yours (ours at a 1500 MHz clock cap, MTP-4, KV pinned to 10.73 GiB/rank):

┌───────────────────────────┬─────────────┬────────────────────────┐
│                           │ your TP=2   │ our TP=3               │
├───────────────────────────┼─────────────┼────────────────────────┤
│ PP @ d0/d6k               │ 1,558–1,564 │ 1,591                  │
├───────────────────────────┼─────────────┼────────────────────────┤
│ PP @ d32–64K              │ 1,477–1,504 │ 1,509                  │
├───────────────────────────┼─────────────┼────────────────────────┤
│ TG c1 @ d64K              │ 32.73       │ 30.8 (≈32 at 2100 MHz) │
├───────────────────────────┼─────────────┼────────────────────────┤
│ TG aggregate, small depth │ 40.24 (c4)  │ 40.5 (c3)              │
├───────────────────────────┼─────────────┼────────────────────────┤
│ KV pool                   │ ~990K       │ 1.51M (4.72M unpinned) │
├───────────────────────────┼─────────────┼────────────────────────┤
│ Weights/rank              │ ~90 GiB     │ 63.8 GiB               │
└───────────────────────────┴─────────────┴────────────────────────┘

Prefill is identical within noise. Decode is identical within noise. The only thing that actually moved is memory.

Why it doesn’t scale

Rough arithmetic on our side: ~18B active params, experts in NVFP4, so a forward pass touches ~11 GB of weights, i.e. ~3.7 GB per rank at TP=3. At GB10’s ~273 GB/s that’s ~14 ms. Measured step time is ~70 ms. Collectives don’t explain the gap either — we measured 37 µs for a 28 KB all-reduce on this ring, so ~90 all-reduces per step (45 layers × 2) is ~3.3 ms.

So roughly 80% of the step is fixed per-step overhead, not per-rank work. Two independent measurements confirm it:

• +40% SM clock buys +4.4% decode (1500 → 2100 MHz: 35.23 → 36.79 tok/s). Compute-bound decode would scale far better than that.
• The same clock change buys −26% on prefill wall clock (64K: 44.5 s → 35.3 s), which is GEMM-bound.

Sharding work across a third rank cannot help something dominated by fixed overhead — and it adds a collective hop. Hence TP=3 ≈ TP=2.

Where the actual headroom is

1. CUDA graphs. Every published recipe for this model, ours included, runs --enforce-eager. On a 45-layer hybrid (KDA + sparse-MLA + MoE) that’s a lot of launch overhead per step, and it is exactly the cost that doesn’t shrink with TP. This is the biggest single lever and we have not tried it — upstream’s 3× repo has a t6-graphs experiment worth reading.
2. Clock. +26% prefill for +1 °C at 2100 MHz. Free if your thermals allow.
3. Chunked-prefill fairness. Our aggregate tg collapses at c2/c3 with long prompts (160K: 34 → 7.5 → 6.1 tok/s). That’s not the fabric — it’s the step budget: max_num_batched_tokens=8192 means one prefill chunk (~5.6 s) consumes the whole step, so anyone already decoding gets one token per chunk. Lowering it to 2048 should largely fix it. The proper knobs (max_num_partial_prefills, long_prefill_token_threshold) aren’t exposed in these day-0 builds.
4. MTP tuning. Our decode swings 32–39 tok/s run to run purely on draft acceptance. Trying 3 vs 4 vs 5 speculative tokens is cheap.
5. Expert parallel. 288 experts ÷ 3 = 96/rank, changing MoE traffic from all-reduce to all-to-all. Untested here.
6. NCCL for small messages. Small all-reduces are latency-bound; NCCL_PROTO=LL/LL128, channel counts, and the mesh plugin for triangle wiring are all unexplored.

TP=3 costs you patches. 64 attention heads, 64 KV heads, 64 KDA heads and MoE intermediate 2048 — none divide by 3. Running it requires padding heads 64→66, MoE 2048→2112, padding_size = lcm(64,3) on the vocab, data-parallel vision encoder (16 heads), four upstream vLLM overlay files, and a mutated on-disk config.json for SpeculativeConfig. We also had to write one more overlay ourselves: on GB10 any request past ~25K tokens killed the engine, because the sparse indexer’s persistent_topk needs more CTAs than a GB10 has occupancy for and its FilteredTopK fallback wants ≥128 KB smem while GB10’s opt-in ceiling is
101,376 B. There’s a generic kernel in the same function that works fine; you just have to catch the error and fall back to it.

Also note the padding is real waste: 66/64 heads and 2112/2048 MoE means ~3% of attention and MoE FLOPs run on zero-weight padding, on every token, forever.

TP=4 costs you nothing. 64 ÷ 4 = 16, 2048 ÷ 4 = 512, 16 vision heads ÷ 4 = 4. No head padding, no loader overlays, no config mutation, no vocab lcm trick. Plus it’s the config where upstream reports the GB10 KV-allocation ceiling dissolving (~50 GiB weights/rank leaves the 1M-token pool room to allocate).

So yes — for this model, 3 is a genuinely awkward middle: it works, we have it serving 512K context with 471K-token retrieval verified and Xid 0, but you pay in patch surface and you get memory rather than speed. If your bottleneck today is context length or KV pressure, the third node solves it. If it’s tokens per second, the third node will disappoint you and the fourth is the one that changes the geometry.

[jc2375]

Hey, we tend to share recipes and such, if you got it running in TP2, it will be useful to the rest of the community if you can share your vllm recipe!

[ifssupyuckdawg]

Thanks for sharing! I have three DGX Sparks currently and will try this out. I’m thinking about getting another one to do a 4x cluster like the below thread but the MikroTik CRS804 is out of stock. It’s supposed to be back in stock in like a month though. I’m considering getting the fourth DGX Spark and just running it separately until I can get a switch.

GLM-5.3-Flash on 4× DGX Spark: ~30–43 tok/s, 1M context, uncensored, multimodal, cuda graphs on,

GLM-5.3-Flash on 4× DGX Spark: ~30–43 tok/s, 1M context, uncensored, multimodal Native FP8. Tensor-parallel 4. CUDA graphs on. Stock vLLM does not boot this model on GB10. This is the recipe that does. Repo (setup, lanes, benches, patches): GitHub - Wpnx330/GLM-5.3-Flash-FP8-4x-DGX-Spark: GLM-5.3-Flash native FP8 on 4x DGX Spark (GB10): occupancy lanes 15@200k / 5@500k / 3@1M, CUDA graphs, multimodal. Followable recipe. · GitHub Tony’s 2-Spark NVFP4 writeup is what made GB10 even possible. We …