You’re right on that:
tool-eval-bench
New local-inference-lab quant: 89/100 — 156/176 points
Previous LibertAIDAI quant: 85/100 — 149/176 points
[ Z.AI FP8 quant: 90/100 ]
Improvement: +4 score, +7 points
Outcomes: 73 pass, 10 partial, 5 fail
Largest category changes: P +3, K +2, O +2; H and I −1 each
By the way, VLLM for this quant needs a slight fix in modelopt.py, because MTP layer are stored under language_model.model.layers.45... but DSv4F fixes that without issues.
update coming soon with DFLASH
I did tool-eval-bench over three models and results are:
| recipe | quant | quality | points | decode tg t/s (c1) | tg c2 | TTFT | deployability | MTP |
|---|---|---|---|---|---|---|---|---|
glm53-flash-nvfp4-lab |
local-inference-lab mixed NVFP4/MXFP8-MTP | 89 | 155/174 | 28.4 | 43.4 | 3.9 s | 70 | ✅ k=3, accept len ~2.9–3.5, ~63–83 % |
glm53-flash-nvfp4-redhat |
RedHatAI W4A4 compressed-tensors | 90 | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) |
glm53-flash-nvfp4 |
LibertAIDAI NVFP4-A16 | 88 | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
Basically, the variant without MTP produced the best, but slowest result.
My find from yesterday was that mtp or dflash plus cuda graphs seriously hit quality, you may want to try disabling cuda graph capture with mtp
That shouldn’t be the case for either MTP nor DFlash if implemented correctly I think, all generated tokens are verified by the main model?
Anyway, I’m not an expert, but I tried to get the model running with acceptable performance and large KV pool for my agentic needs, and I am now at:
I used the local-inference-lab model but manually replaced the MXFP8 drafter with the NVFP4 drafter from LibertAIDAI.
I also:
GMU 0.89
disabled video
maxlen 262,144
batched-token limit 2,048
eager mode
mtp3
maxseq4 (diminishing returns, not more efficient than queuing additional requests)
FP8 KV
Marlin
I still need to see how the quantized drafter and MTP3 holds up, but it’s really hard to measure due to warming and workload variations.
There is a big gap between theory and practice. Very complex algorithms, whole graph thing is non-deterministic, quantization introduces noise too.
Follow-up: @0rand you was right - disabling CUDA graphs with MTP. A/B with identical seed/flags (tool-eval-bench --seed 42 --hardmode --perf --concurrency 1,2) on the lab variant:
| recipe | quant | graphs | quality | points | decode tg t/s (c1) | tg c2 | TTFT | deployability | MTP |
|---|---|---|---|---|---|---|---|---|---|
glm53-flash-nvfp4-lab + --enforce-eager |
local-inference-lab mixed NVFP4/MXFP8-MTP | ❌ | 92 | 162/176 | 28.4 | 40.5 | 5.6 s | 72 | ✅ k=3, accept len ~3.2, ~72 % |
glm53-flash-nvfp4-redhat |
RedHatAI W4A4 compressed-tensors | ✅ | 90 | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP in checkpoint) |
glm53-flash-nvfp4-lab |
local-inference-lab mixed NVFP4/MXFP8-MTP | ✅ | 89 | 155/174 | 28.4 | 43.4 | 3.9 s | 70 | ✅ k=3, accept len ~2.9–3.5 |
glm53-flash-nvfp4 |
LibertAIDAI NVFP4-A16 | ✅ | 88 | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
Glad to help, mate!
FYU - tried this party trick with DS4F DSpark - lost perf, did not help quality. But it did help a LOT with Qwen 3.8 Flash Next DFlash2 and MTP so worth a try for those cases.
PS now you convinced me to try GLM beast too, lol
I did not mention it, but I see other recipes use 256K context. I have 512K context and 2x concurrency (visible in my test command).
Nice, that is what I am looking for - 512k is my staple, 1M is more aspirational, 300-500 is workhorse.
btw from our man Luke - new quant just hit the wires
Main model:
Routed Expert Projections - NVFP4 (calibrated, with input scales for A4)
Attention Projections: MXFP8
Shared Experts: MXFP8
MTP:
Routed Expert Projections - NVFP4 (A16 only)
Attention Projections: MXFP8
Shared Experts: MXFP8
Will try that, to contribute to the gang, like my old boss said - sometimes you need to bend over and take one for the team :D
Quick question - did you use JovialJudgement VLLM fork from Luke to run his quant? If so - built from source or there is some wheel floating around? Recipe to start from?
So this is a fresh and different take on GLM-5.3 flash. I’ll try it out later or tomorrow. Would be great to hear your experience if you get to it before me.
now we talking - first dflash and 60 toks, more cache room. lets try that instead!
Here you are:
The recipe is for my sm121 launcher, but I think idea is more or less clear.
Model loaded… multimodal confirmed working [PLACEHOLDER - will edit with benchmarks]
(Worker_TP0 pid=775) INFO 08-28 11:34:19 [gpu_worker.py:790] Free memory on device (110.2/121.69 GiB) on startup. Desired GPU memory utilization is (0.87, 105.87 GiB). Actual usage is 86.46 GiB for consumed memory (weights + non-torch), 4.18 GiB for peak activation, and 0.52 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=15639670334` (14.57 GiB) to fit into requested memory, or `--kv-cache-memory=20286027776` (18.89 GiB) to fully utilize gpu memory. Current kv cache memory in use is 15.23 GiB.
Load complete. Going to just give it a greeting and a test image, then run benchmarks.
Quite a wait for a response, the image check was OK, and the same image I gave to Qwen3.8-Flash-Next - the description was not quite as good as Qwen’s but certainly I would rate it as “Good”, I have no doubt that if I were to request more info or modify the original prompt to request more detailed information that it would probably have improved it.
No --base-url provided, scanning localhost…
✓ Auto-discovered vLLM at http://localhost:8000
Detected backend: vLLM
🔧 Tool-Call Benchmark
Server: http://localhost:8000
Querying http://localhost:8000/v1/models … ✓ /root/.cache/huggingface/hub/models--brandonmusic--GLM-5.3-Flash-tr3-4bpw/snapshots/5ab363a8dcf6405955fd5f99671e01a1c9fb124b (alias:
GLM-5.3-Flash-EXL3)
✓ Warm-up complete (361 ms)
🔍 Engine: vLLM 0.1.dev20051+g487ecf187
🔤 Tokenizer: brandonmusic/GLM-5.3-Flash-tr3-4bpw (HuggingFace cache)
╭─────────────────────────────────────────────── ⚡ llama-benchy Throughput Benchmark ───────────────────────────────────────────────╮
│ /root/.cache/huggingface/hub/models--brandonmusic--GLM-5.3-Flash-tr3-4bpw/snapshots/5ab363a8dcf6405955fd5f99671e01a1c9fb124b │
│ pp=[2048] tg=[128] depth=[0, 4096, 8192] concurrency=[1, 2, 4] runs=3 latency=generation │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 27/27 0:11:48
llama-benchy 0.4.0
Estimated latency: 208.1 ms
llama-benchy Results
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Test ┃ c ┃ pp t/s ┃ tg t/s ┃ TTFT (ms) ┃ Total (ms) ┃ Tokens ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ pp2048 tg128 @ d0 │ c1 │ 667 │ 31.8 │ 3,301 │ 7,115 │ 2048+128 │
│ pp2048 tg128 @ d0 │ c2 │ 585 │ 36.2 │ 6,520 │ 11,886 │ 2048+128 │
│ pp2048 tg128 @ d0 │ c4 │ 434 │ 28.8 │ 10,691 │ 17,106 │ 2048+128 │
│ pp2048 tg128 @ d4096 │ c1 │ 751 │ 35.2 │ 8,405 │ 11,835 │ 2048+128 │
│ pp2048 tg128 @ d4096 │ c2 │ 757 │ 22.4 │ 13,262 │ 20,163 │ 2048+128 │
│ pp2048 tg128 @ d4096 │ c4 │ 654 │ 15.3 │ 23,723 │ 31,342 │ 2048+128 │
│ pp2048 tg128 @ d8192 │ c1 │ 779 │ 31.6 │ 13,370 │ 17,207 │ 2048+128 │
│ pp2048 tg128 @ d8192 │ c2 │ 727 │ 14.1 │ 21,819 │ 29,650 │ 2048+128 │
│ pp2048 tg128 @ d8192 │ c4 │ 696 │ 10.5 │ 36,665 │ 45,758 │ 2048+128 │
└─────────────────────────────────┴─────────┴────────────────┴────────────────┴──────────────────┴─────────────────┴─────────────────┘
╭────────────────────────────────────────────────────── 🏆 Benchmark Complete ───────────────────────────────────────────────────────╮
│ │
│ Model: │
│ /root/.cache/huggingface/hub/models--brandonmusic--GLM-5.3-Flash-tr3-4bpw/snapshots/5ab363a8dcf6405955fd5f99671e01a1c9fb124b │
│ Score: 90 / 100 │
│ Rating: ★★★★★ Excellent │
│ Benchmark: tool-eval-bench v2.6.1.dev25+g4365b9031 │
│ Engine: vLLM 0.1.dev20051+g487ecf187 │
│ Max context: 900,000 tokens │
│ │
│ ✅ 74 passed ⚠️ 10 partial ❌ 4 failed │
│ Points: 158/176 │
│ │
│ Quality: 90/100 │
│ Responsiveness: 22/100 (median turn: 7.0s) │
│ Deployability: 70/100 (α=0.7) │
│ Weakest: M Autonomous Planning (67%) │
│ │
│ Completed in 2636.4s │
│ │
│ 📊 Token Usage: │
│ Total: 543,496 tokens │ Efficiency: 0.3 pts/1K tokens │
│ │
│ 🛡️ SAFETY WARNINGS (1): │
│ ⚠ TC-43 (Omitted Required Parameter): Called web_search with an empty query — violated required parameter constraint. │
│ │
│ ⚡ Throughput: │
│ Single: 779 pp t/s │ 35.2 tg t/s │ TTFT 8,405ms │
│ c2: 757 pp t/s │ 36.2 tg t/s │
│ c4: 696 pp t/s │ 28.8 tg t/s │
│ │
│ ── How this score is calculated ── │
│ • Each scenario: pass=2pt, partial=1pt, fail=0pt │
│ • Category %: earned / max per category │
│ • Final score: (total points / max points) × 100 │
│ • Deployability: 0.7×quality + 0.3×responsiveness │
│ • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
I’ve run this with:
tool-eval-bench --hardmode --seed 42 --parallel 1 --max-turns 32 --backend-kwargs '{"chat_template_kwargs": {"thinking": true, "reasoning_effort":"max"},"temperature": 0.8,"top_p":0.25}' --perf --trials 3
So, after three runs:
╭─────────────────────────────────────────────────────── 📊 Trial Statistics ────────────────────────────────────────────────────────╮
│ │
│ Trials: 3 │
│ Score: 88.7 ± 1.2 / 100 │
│ Median: 88.0 │
│ 95% CI: [88.0, 90.0] │
│ Points: 155.7 ± 2.1 │
│ │
│ Pass@3: 86.4% (capability ceiling) │
│ Pass^3: 76.1% (reliability floor) │
│ ⚠ Gap: 10.3pp (high variance — consistency issue) │
│ │
│ Categories with variance: │
│ A Tool Selection: 94% ± 9.8% │
│ G Structured Reasoning: 94% ± 9.8% │
│ I Context & State: 82% ± 2.9% │
│ K Safety & Boundaries: 81% ± 4.0% │
│ O Structured Output: 83% ± 16.5% │
│ P Hard Mode: 92% ± 3.0% │
│ │
│ ⚡ 10 unstable scenario(s): │
│ TC-03: 1.7 ± 0.6 (2,1,2) │
│ TC-21: 1.7 ± 0.6 (2,1,2) │
│ TC-33: 1.3 ± 1.1 (2,2,0) │
│ TC-46: 1.3 ± 0.6 (1,1,2) │
│ TC-58: 1.7 ± 0.6 (2,1,2) │
│ TC-67: 1.3 ± 0.6 (2,1,1) │
│ TC-68: 1.3 ± 1.1 (2,2,0) │
│ TC-69: 1.3 ± 0.6 (2,1,1) │
│ TC-82: 1.3 ± 0.6 (1,1,2) │
│ TC-85: 0.7 ± 0.6 (0,1,1) │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Interesting that of those that were partial or failed, on some runs it scored correctly. For me this means if it gets it wrong, with further context or prompting it will eventually get it right.
I will just repeat a single test without the --backend-kwargs now.
Big respect for full transparency including sampling parameters!
That was fast! Thanks for sharing.
High variance is very typical for any high temp setup. Some runs will be higher, some lower. Lower temperature (0 is closest to deterministic but not fully - still random noise in floating math rounding) will get a similar base with much lower variance. Up to you to decide which to use. 0 temp often lead to loops in harder reasoning, so often a mid-ground is preferred.
Agreed, can understand how that would work. Would just be important that in a more long lived goal-orientated setting with more context I get the feeling it would be quite workable.
Do I consider it a contender yet? Well… it is a bit slow, and DSV4F is doing so well, I would say this isn’t something I’m going to delete off disk, but it might not live constantly in RAM. It is something I might turn to as a substitute when DSV4F seems to be flagging a little. Plus it is multimodal!! Mustn’t forget that. Might make it an end-of-day debugger !
There also have been quite a few commits to the repo since I spun this up, so I’m likely to try a repeat build over the weekend.
No performance improvements there, only stop reasoning fix.
The rest is different warmup and monitors, caching boot shapes.