Same! TP4 until a TP2 quant I like drops.
Two DGX Spark results: GLM-5.3-Flash-NVFP4 at 23 tok/s and 200k context
Following up with measurements for the 195 GB LibertAIDAI/GLM-5.3-Flash-NVFP4 quant: it fits on two DGX Sparks with vision and MTP enabled. The earlier 20â25 tok/s estimate was accurate.
Configuration
- 2Ă DGX Spark/GB10, TP=2 over the 100 Gb fabric
- vLLM, native SM12x extension, CUDA 13, eager execution
- Marlin routed-MoE backend, FP8 MLA KV cache
- MTP with five speculative tokens
- 200,000-token model limit, 2,048-token prefill batches, GPU memory utilization 0.90
- InstantTensor loading; vLLM reports 91.21 GiB loaded per rank in approximately 103 seconds
- Vision, reasoning and automatic tool calling enabled
Performance
Benchmark: 161 prompt tokens, 256 forced output tokens, temperature 0.
| Configuration | c1 output | c1 TTFT | c5 aggregate output | Improvement over base |
|---|---|---|---|---|
| Base extension + Marlin | 7.96 tok/s | 0.515 s | 19.37 tok/s | Baseline |
| Native SM12x + Marlin | 14.48 tok/s | 0.398 s | 50.69 tok/s | +82% c1 / +162% c5 |
| Native SM12x + Marlin + MTP=5 | 23.08 tok/s | 0.439 s | 60.68 tok/s | +190% c1 / +213% c5 |
Tool-use quality
tool-eval-bench, standard 69-scenario suite, seed 42, temperature 0, thinking disabled:
- 86/100 â â â â â Good (118/138 points; 54 pass, 10 partial, 5 fail; 0% infrastructure errors)
- 100% in tool selection, parameter precision, restraint, error recovery, localization, structured reasoning, toolset scale and creative composition
- Weakest category: autonomous planning at 50%; safety was 77%, with failures on an empty required argument and an injected fake system message
- Median turn latency: 6.2 seconds; full run: 30m50s
- During the run: 23.4 tok/s mean generation rate, 44.3 tok/s peak, 3,793 prompt tok/s peak, 16.7% peak KV use and zero queued requests
- MTP accepted 51.3% of drafted tokens, with a mean acceptance length of 3.57 tokens
KV-cache capacity
With MTP=5, vLLM allocates 9.14/9.23 GiB on the two TP ranksâapproximately 18.4 GiB physicallyâand reports 848,717 logical token slots. That is 4.24Ă 200k contexts.
--max-num-seqs 5 only permits five requests when their combined resident context stays below 848,717 tokens. It does not fit five full 200k contexts; the guaranteed limit is four full 200k sessions.
Disabling MTP increases the reported logical cache to 1,385,714 tokens, or 6.93Ă 200k, but reduces output performance to 14.48 tok/s at c1 and 50.69 tok/s aggregate at c5. Use MTP=5 for speed or disable it when maximum long-context concurrency matters more.
Main challenges and fixes
| Problem | Fix |
|---|---|
| Native CUTLASS MoE produced syntactically valid but corrupted output, despite isolated kernel tests passing. | Keep the native SM12x extension for dense FP4 operations, but force routed experts through --moe-backend marlin. |
| Initial multimodal requests failed with âAt most 0 image(s)â. | Update the model chat template and launch with --limit-mm-per-prompt '{"image":8,"video":1}' --mm-processor-cache-type lru. |
| MTP improves decoding but consumes memory and the DSA backend cannot use fused multi-step draft metadata. | Keep MTP=5 for the throughput profile; disable MTP for the larger KV-cache profile. |
| Raising GPU-memory utilization leaves too little unified-memory headroom for long prefills. | Keep utilization at 0.90 and prefill batches at 2,048. An actual 180,013-token request completed successfully with these settings. |
Text generation, image input, tool calling, c5 traffic and an actual 180,013-token prompt all passed after these changes.
Thatâs good kv cache wise, actually. But why your Ai thinks model has 200k session limit? Itâs 1m. Itâs slow, but surely dspark will appear
In 2 sparks it will be NVFP4 which will dumb it down to something like lower Opus 4.x level.
In my experience Running Q2 Deepseek v4 Flash 0731 dumbs it down from solid Opus 4.8 to something like Sonnet 4.5 ish
â
I had faith in full version after looking at Q2, and my bet was the cost of a second Spark. No regrets :)
i think the trade off is not as big as you think, of course fp8 would be better but this performance will still play out everything else, only speed it my concern, when i can get this to 40tk prose and higher with code this will be massive⊠hoping for some dflash/dspark
I told it to run max 200k context and get as many concurrent sessions as possible :) did some testing late last night on DCP=2 but had to drop MTP for that, 2.3M KV total means -c2 @1M at 30 tok/s agg
Iâm a little disappointed, I tested NVFP4 against openrouter fp8 with tool-eval-benchv2.6.1.dev18+gcad5bfb5b cad5bfb-dirtyin hard mode and seed 42.
NVFP4: 85 / 100
FP8: 90 / 100
Iâm still running HumanEval as a quick follow up benchmark but not the results I was hoping for.
Thatâs crazy, but we will get better quants, kv Cache size with nvfp4 and MTP is just over 600K in my current tests, so we need that to run 5.3 Flash for prod env
we need a good autoround quant :)
A real one like LukeAlonso needs to step in and make a quant for us to work off of, I just think the LibertAI one is kind of low quality, doesnât provide kl-divergence.
These metrics from the model card kinda proves this is not a high quality quant:
Round-trip: per-expert cosine â 0.99665, relative error â 0.0925 vs the BF16 source.
For reference, my INT4 Muse Glimmer 30b quant I posted has a kl-divergence of 0.001 which should be standard for a good 4-bit, FP4 or INT4.
Is there just something wrong with how GLM 5.3 Flash and the quantization libraries interact? I would expect way more quantizations of the model to be out like they are for Qwen 3.8 Flash Next
Also would still like to see MXFP4 from the AMD huggingface account or someone else with AMD Quark or something.
HumanEval
| Endpoint | Pass@1 | Errors | Concurrency | Wall time |
|---|---|---|---|---|
| Local NVFP4 | 160/164 = 97.56% | 0 | 4 | 4,639.9 s |
| OpenRouter FP8 | 159/164 = 96.95% | 0 | 10 | 546.1 s |
Local NVFP4 led by one problem / 0.61 percentage pointsâeffectively quality parity, with no observed NVFP4 coding-quality deficit.
All 164 remote responses reported Z.AI, and OpenRouterâs catalog maps that route to FP8; no fallback occurred during HumanEval.
Hmm, so what now?
Whether or not it arrives at the correct answer on a benchmark is not the same as token-for-token parity. Even if it does similarly on a benchmark, 0.99665 cosine similarity is still not particularly good for FP4
The LibertAI quant is still good enough for a POC, for inference compatibility and speed testing. Iâm just countering the earlier member discounting NVFP4 because of this particular quantâs accuracy. Better quants will be created is all Iâm saying, we should be targeting coherence and speed with this, donât worry too much about exact benchmark results of this vs FP8 right now.
LukeAlonso NVFP4
Havenât even looked at the model card yet but I know my boy Luke made another hit. Instant download from me
His MiniMax M2.7 NVFP4 was higher quality than the official Nvidia one
That should be good but 199GB on 2 sparks is literally asking for OOM :(
Maybe not, I have with other weights 1M kv cache achieved, doable but close haha
Last I checked, HF posts sizes in base 10, not base 2. 199GB = 186GiB. htop uses the latter.
its up
Do you plan to optimize further ?
There are also RedHat quants, going to test those from Luke and these.