GLM-5.3-Flash: 320B total parameters / 18B active

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
651

and this is the case ;)

652

can you have a look into that? GitHub - technigmaai/glm-5.3-flash-gb10-tp2-native: GLM-5.3-Flash NVFP4 on two NVIDIA GB10 nodes: native TP2, one container per node, display-backed KV and image input Ā· GitHub

653

Uh, what is ā€œCanadaā€ here. Iā€™ve only been following this loosely lately and Iā€™m really confused. Would love trying it as Iā€™m mostly interested in accuracy as well.

654

Yeah, well Iā€™ve been following it, and yet this is the first Iā€™ve heard of itā€¦ I am perplexed tbh.

Iā€™m going to ask that you adopt my ā€œRochdaleā€ fork istead:

655

HF: canada-quant/GLM-5.3-Flash-W4A16-MTP

656

I would totally be down to try it. Iā€™ve hit a slight pause in my months long document processing task, so it would be a good time. The problem is that I do want to still keep a copy of the old Entrpi variant around, but my Spark(s) barely has/have any space left, so Iā€™m gonna need to clear a lot out somehow, and I feel like Iā€™ve looked under the cushion in almost every couch at this point. My poor 1TB drive has been thoroughly abused on this thing.

Question, does this still keep all of the usual OpenAI endpoints, so I can query it with curl and Open WebUI like normal? Supports tool calling fine, etc? Have seen people complain about that. Does it support the ā€œreasoning_effortā€ flag normally?

Otherwise yeah Iā€™m going to definitely see if I can get it up and running, thanks.

I see, what engine are you using to run it? Or is it just on the HF page? Thanks.

657

sorry, forgot to mention:
eugr/spark-vllm-b12x:nightly-20260927 (vLLM 0.1.dev21510+g1794dcf18, digest sha256:42f762c0...84cd, native MTP

658

Hi all, weā€™re the canada-quant folks behind canada-quant/GLM-5.3-Flash-W4A16-MTP. Thanks @jetspark for the recommendation and the bits-per-byte numbers. Your component table matches how we built it: only the 36,288 routed-expert GEMMs are INT4 (GPTQ, symmetric, group size 128). Attention including the DSA indexer, dense layers 0ā€“2, shared experts, router, embeddings/lm_head, the vision tower and the MTP layer all stay BF16.

One difference being the b12x nightly with native MTP isnā€™t the stack we validate on. Our Spark recipe uses the drafter we built:

The drafter matters most. On our 500-prompt holdout, the built-in MTP head accepts about 2.4ā€“2.5 tokens per step. DFlash2-G accepts 3.68 at K=7; incoaiā€™s reference drafter measured 3.63 on the same hardware.

What our recipe measures on 2Ɨ Spark (llama-benchy pp2048/tg128, single stream):

context depth 0 4K 65K 100K
decode t/s 31.4 Ā±3.5 30.7 15.3 10.0*

*The 100K cell is from the 09-26 grid run; the rest are from a quiet re-run on 09-28. The depth-0 cell is tg32.

  • Context: with DFlash2-G we serve 262K by default (366,749-token KV pool at 8 GiB fp8); the llama-benchy numbers above were run at 800K.
  • AIME 2026 (n=120, max thinking, 131K budget): 85.0% (102/120), vs 80.0% for EXL3 under the same config on the same pair.

Cold boot on our image is 6ā€“10 min, so your 145 s with instanttensor + read_ahead_kb beats ours. Nice find.

@jetspark, if you can run the same llama-benchy grid on your b12x + MTP setup, weā€™d love to see it next to ours. We havenā€™t measured our weights on b12x (with MTP or our drafter), and it may well be faster.

For anyone using the b12x build: this checkpoint needs the ignore-list workaround from eugr/spark-vllm-docker#403. The broad self_attn regex is safe here because nothing under self_attn is quantized. The narrow version is re:.*\.self_attn\.(f_a_proj|f_b_proj|in_proj_qkvgfab)$.

Weā€™ve been running it 100% flat-out for days generating synthetic data, with great results.

659

What would be the current most recommended GLM recipe for 2 sparks?

660

I know I said I mostly cared about quality, but 10 t/s at higher context is oof levels of slow, I can only assume it goes downhill from there, too. I guess I will try out these TensorRT ones first.

661

here we go:

context depth 0 4K 65K 100K
decode t/s 31.6 Ā±3.4 29.8 Ā±3.3 25.4 Ā±2.2 31.0 Ā±0.7

NOTE: gpu clock = 1700

cmd 
tool-eval-bench   --base-url http://jetspark01:8045 \
   --perf-only --tokenizer canada-quant/GLM-5.3-Flash-W4A16-MTP \
   --benchy-runs 2 --depth 0,4000,65000,100000 \
   --concurrency 1,2
live monitoring

image
image1611Ɨ1864 503 KB

P.S. thanks for your quant :)

UPDATE:
I tried the canada-quant drafter hoping to free up some memory, since itā€™s way smaller than the built-in MTP head: 6 GB vs 15 GB. Turns out it does the opposite. Itā€™s 8 full layers, and each one keeps its own uncompressed cache for the whole context, so on long chats that cache eats way more than the smaller weights save. My usable context dropped from about 930K tokens to 330K, and 512K didnā€™t fit anymore.
On top of that it was 20-30% slower in non English, though faster on English code. So Iā€™m sticking with MTP.

662

Well holy moly, this thing is indeed really fast. Or at least it feels fast. Unfortunately TensorFold doesnā€™t seem to report ongoing generation numbers, so itā€™s hard to get any idea of how fast itā€™s going on average per second or something. Will now start trying it on my actual dataset. Iā€™m just hoping the output quality has not degraded. For some reason people found that MiaAI had issues with not scoring as high as entrpiā€™s engine on code bench testing so Iā€™m a bit wary about the quality. We shall see.

663

I switched from vllm to tensorfold without any modification and opencode and openwebui continued to just work.

664

Something thatā€™s a bit weird: why does this default to a default max_tokens of ~32.7k?

Thatā€™s just really low. The kv cache size is over 1 million, so itā€™s just odd that it stops 1/30th of using it. I mean I guess thatā€™s to make it easier to process 1m size contexts safely, so I guess thatā€™s made for agentic work where it might need to keep a lot of previous work in context but each output could be low?

Either way for now I just bumped it to 1m for my ~120-150k token prompt size, but I guess to be safe I would set it to around 800k to give enough room for reasoning and then output. Itā€™s kind of weird that the engine asks me this though I guess. VLLM just chugged along till each payload was done processing. I just noticed this one was stopping at almost exactly 100k chars of output every time which was mildly annoying because it was just silently botching every single request and I didnā€™t even know it.

Otherwise, still seems to be quite fast. I guess I will leave it open for a few weeks to see if it can just keep going like entrpiā€™s engine did.

665

Have tested and found that on the 9GiB there is early OOM. Shirnk KV cache to 7730941132 and it stay megastable during long run.

And this is the new king.

Here are the results:

GLM-5.3-Flash on 2ƗDGX Spark ā€” variant T: technigmaai native TP2 (C6)

Follow-up to the K/K2/K-6144 series: this run moves from the kindlingai Mentat/Ray
serving stack to the technigmaai glm-5.3-flash-gb10-tp2-native recipe (@24ba8fb) ā€”
native vLLM multiprocess TP2, no Mentat/Ray, display-reserved KV carveout (+1.79 GiB), and
same pinned model bytes as everything before: nvidia/GLM-5.3-Flash-NVFP4 (NVFP4 W4A4
routed experts, BF16 attention/shared/MTP) + incoai/GLM-5.3-Flash-DFlash2 (k=7) speculative.
Weight snapshots were reused from our kindlingai run (identical vLLM build) ā€” zero re-download,
zero rebuild of ~90 GiB snapshots.

Profile (C6): MAX_NUM_BATCHED_TOKENS 6144 Ā· MAX_NUM_SEQS 6 Ā· KV_CACHE_MEMORY
7.2 GiB (7730941132 ā€” matching our proven kindlingai pin, NOT the recipeā€™s riskier 9 GiB
default, see OOM note) Ā· max len 1,047,552 Ā· fp8_e4m3 KV Ā· DFlash2 k=7 Ā· TP2 over RoCE
(.23 head + .25 worker).

Headline ā€” wall-time and score/hour first (our decision metric):
T tool-eval high 278 pts/hr (ties K2, beats K 274 and K-6144 257) and
max 205 pts/hr (beats K-6144 185, K2 207 about equal) ā€” at quality high 92.0 / max 96.0,
the highest max score of the whole series (K 95.67, K-6144 95.0, K2 92.0).
Total wall for the 6-run tool-eval set = ~144 min (vs K-6144 157, K 147.9, K2 139.6).
Cold prefill +6% over K-6144 (2678 vs 2522 t/s), TTFT@262k āˆ’9% (100.4 vs 110.5 s).

Method (identical harness to the whole series): cold-prefill+gen sweep, 64 points,
4096 ā†’ 262 144 tokens, UNIQUE prompt per size (seed = size ā‡’ radix cold), streamed SSE
first-byte TTFT, forced 256-token gen; tool-eval-bench 3Ɨ high + 3Ɨ max, seed 42, hardmode,
Ī± 0.7, per-request reasoning_effort.

ā± Wall-time & utility (the decision metric)

Config stack high pts/hr high time/run high max pts/hr max time/run max total wall
T (this) technigmaai native TP2 92.0Ā±0.0 278 19.8 min 96.0Ā±0.0 šŸ† 205 28.1 min ~144 min
K-6144 kindlingai TP2 b.6144 92.0Ā±0.0 257 21.5 min 95.0Ā±0.0 185 30.8 min 157 min
K kindlingai TP2 b.4096 92.0Ā±0.0 274 20.2 min 95.67Ā±0.58 196 29.2 min 147.9 min
K2 Mia TensorFold Ɨ4 92.0Ā±0.0 278 19.9 min 92.0Ā±0.0 207 26.7 min 139.6 min
D eugr b12x 91.33Ā±0.58 230 ā€” 92.0Ā±1.0 165 ā€” ā€”

(all n=3 except historical A/B; equal quality on high ā€” the knob/topology moves only speed.)

ā± Cold-prefill sweep (64/64 ok, zero OOM on every config)

Config pp avg t/s gen avg t/s TTFT avg TTFT@262k KV pool (tokens)
T (new) 2678 42.7 49.8 s 100.4 s 1,215,513
K-6144 2522 38.2 53.7 s 110.5 s 1,165,254
K 2296 42.7 59.6 s 129.8 s 1,169,491
K2 Ā· TensorFold 1620 59.0 85.3 s 181.4 s 2,033,664

šŸ“ˆ Cold prefill per size (t/s) ā€” T vs the field

size K (4096) K-6144 T Ī” T/K-6144 K2
4k 2411 2610 2571 āˆ’1% 1649
32k 2360 2681 2627 āˆ’2% 1765
64k 2366 2501 2655 +6% 1729
128k 2100 2552 2737 +7% ā€”
196k 2271 2497 2672 +7% ā€”
262k 2019 2372 2610 +10% 1445

T is the flattest prefill curve of the series: 2453ā€“2784 t/s across the whole 4kā€“262k
range (median 2676), no sag toward the long end. At 262k it beats K-6144 by +10% and
TensorFold by +81%. The only T losses are small and in the 4kā€“32k cold start zone (constant
multiprocess overhead); from 40k up it wins every point (55/64 across prefill & TTFT).

āš— Generation (t/s, forced 256-tok single-stream cold)

K K-6144 T K2
avg over grid 42.7 38.2 42.7 59.0
@262k 39.1 38.8 40.4 51.8

T restores the generation throughput that the kindlingai 6144 bump gave away (42.7 ā‰ˆ K,
above K-6144ā€™s 38.2) while keeping the 6144-sized prefill win ā€” the best of both. Sprint
decode still served by DFlash2 k=7.

šŸ’¾ KV/context implications

  • KV pool 7.2 GiB = 1,215,513 tokens (fp8_e4m3, block 2304, LBHNC). Same pin as kindlingai
    ā‡’ same 2.33Ɨ headroom over the 500k window.
  • OOM note: the technigmaai C6 recipe ships KV 9 GiB (9663676416), which on these
    unified-memory boxes overflows real host RAM at ~155k (vLLM RuntimeError: cancelled ā†’
    EngineDeadError, client sees ConnectionError; boot log warns 130 GiB needed, 117 free).
    Pinning 7.2 GiB ā€” the value proven across K/K-6144 ā€” eliminates it. 300k sanity re-check
    passes clean, sweep completes 64/64, tool-eval 6/6 with zero swap pressure.
  • TensorFold still holds the biggest KV headroom (2.03M) but pays for it in loaded-cache
    parallel decode (llama-benchy d8192c4 ā‰ˆ 17 vs 26-28 t/s on kindlingai/T-style stacks).

šŸŽÆ Tool-eval details (n=3, seed 42, hardmode, per-request reasoning_effort)

variant high max spec-eff high/max (t/s)
T 92.0 Ā± 0.0 96.0 Ā± 0.0 63.4 / 65.1
K-6144 92.0 Ā± 0.0 95.0 Ā± 0.0 61.1 / 60.5
K 92.0 Ā± 0.0 95.67 Ā± 0.58 ā€”
K2 92.0 Ā± 0.0 92.0 Ā± 0.0 ā€”

Tā€™s max-effort quality (96.0) is the best measured on this farm, at the fastest high-effort
time (19.8 min) ā€” and its spec-decoding efficiency is up across the board (63.4/65.1 vs
K-6144ā€™s 61.1/60.5 t/s).

Takeaways

  • T is the new leader on both axes we care about: best max quality (96.0) and best
    high throughput-per-hour (278 pts/hr, tied with K2). No other config has done both.
  • Native TP2 (no Mentat/Ray) is strictly better than kindlingaiā€™s Mentat flavor at equal
    knobs    : +6% prefill, āˆ’9% TTFT@262k, +12% gen, +3 pts/hr high ā€” on byte-identical weights,
    no code changes, seed-reused snapshots.
  • KV pin portability warning: take the recipeā€™s KV default with skepticism on
    unified-memory 2ƗGB10; reuse your previously-proven pin and verify with a long sanity
    prompt before the full grid.
  • If you are decode-bound with short prompts, TensorFold K2 (59 t/s gen) is still the pick;
    for long-prompt/RAG/agentic ā€” T is now the fastest, highest-quality option.

Files / reproducibility

Bottom line: swapping the serving topologie (Mentat/Ray ā†’ native multiprocess TP2) on the
same model bytes, same batch-6144 profile, buys +6% prefill, āˆ’9% TTFT@262k, restores gen to
K levels, and posts the best max-effort score (96.0) of the whole Aā†’T series ā€” with zero
re-downloads and no OOM once KV is pinned to the proven 7.2 GiB.

666

Have you tried Gilesā€™ engine above?

image
image820Ɨ490 29.1 KB

They are showing average prefill of around 1.85-2k. Generation speed seems to still be about the same.

Relinking engine for completeness:

Just wondering if youā€™re using an outdated tensorfold variant since your prefill numbers seem low. Overall, looking at your own charts, not sure if you can call it a clean win when Tensorfold variants are doing 38-39% faster gen though. I guess it depends on what you need the model to do.

667

I have used latest Miaā€™s variant on this Friday. Will try to check in the night gilescoā€™s version.

668

Itā€™s output tokens, not max session length

669

I know, but that includes reasoning. On entrpiā€™s engine, my task could reason for over an hour per input. I guess my issue is that I have to (like actually have to) manage output length as well as session length on this manually?

Anyhow:

[tensorfold] done req-a786c249a3a6 prompt=10 cached=0 thinking=False tokens=2 sha=694aab7656bc finish=stop tok/s=22.4 ttft=0.12s prefill=0.12s rounds=1 accepted=0/1
[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 -
[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 -
[tensorfold] done req-7e5ddeb869b2 prompt=124425 cached=0 thinking=True tokens=168270 sha=91774f45078c finish=stop tok/s=54.8 ttft=77.71s prefill=77.58s rounds=39095 accepted=129174/215637
[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 -
[tensorfold] done req-6aca84fc73fc prompt=112713 cached=0 thinking=True tokens=700500 sha=a698a468ee60 finish=length tok/s=82.7 ttft=69.76s prefill=69.65s rounds=47633 accepted=652866/702503
[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 -
[tensorfold] done req-e994cdb280b3 prompt=112713 cached=112704 thinking=True tokens=700500 sha=a698a468ee60 finish=length tok/s=83.2 ttft=0.13s prefill=0.10s rounds=47633 accepted=652866/702503
[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 -
[tensorfold] done req-d609df6aee1a prompt=112713 cached=112704 thinking=True tokens=700500 sha=a698a468ee60 finish=length tok/s=82.7 ttft=0.13s prefill=0.10s rounds=47633 accepted=652866/702503
[tensorfold] 127.0.0.1 "POST /v1/chat/completions HTTP/1.1" 200 -

I woke up to find out that it hadnā€™t completed a single document outside of the first one. It proceeded to do long-form looping for the entire rest of the time and didnā€™t finish a single one.

grep '12049362 has' glm_logs_21.log |wc -l
510

Iā€™ll restart to see if it was just a bad scenario exacerbated by repeated cache hits which then caused it to do that over and over againā€¦ but it was very clearly looping to the exact same point multiple times. Even if it was going fast, I would have also finished multiple documents by that point on the other engine without the looping. To be fair, itā€™s a slightly tweaked version of my prompt meant to try to find something different, so that can easily cause looping.

670

Just increase to the size you need I guess. It was working for 8 hours non stop today running datamining on 4 different computers for me today (80 cores total to parallel massive work corpus)and building code, mending it, documenting, sending telegram notifications. No looping, no going crazy. BTW MediaLLMProxy has a solution for long reasoning and looping - check it out in my GitHub.