Hi — I’ve been running this recipe (private deployment, both Sparks) since
late August. This thread never got an accuracy discussion, so here’s a
measurement report: quality band, a controlled scheduler A/B, a
thinking-effort comparison, the config deltas I run vs. the recipe as
shipped, and a component-availability warning for anyone rebuilding. It’s
results, not a recipe release. Given it’s apparent stability, accuracy, and large
≈5 M-token pool, I’m surprised there’s not been more interest.
Acknowledgements. @emihuang for the recipe — the EXL3/FP8 profile, the
MXFP8 drafter wiring, and defaulting reasoning effort to max were all
load-bearing for everything below. MiaAI-Lab for the kit underneath and
the EXL3/TR3 4bpw artifact itself. The tool-eval-bench maintainers - it really
helps give a comparison basis for these models.
Configuration as run — three deliberate deltas from the recipe
as-shipped, everything else per the exl3-fp8-dcp2 profile verbatim:
| delta |
mine |
recipe ships |
why |
| speculative K |
fixed K=7, adaptive-K off |
canonical entry defaults to adaptive ema |
A/B’d on my rig, fixed won |
| host posture |
swappiness 60 while serving (0 at rest) |
swappiness 0 always |
two host wedges at 0 with NVRM OOM — the window is the valve |
| model mount |
/models/glm-5.3-flash-exl3-4bpw |
/models/glm-5.3-flash-nvfp4 |
the original path name lies about the contents; I renamed so the engine self-reports honestly |
One identification note for anyone comparing: trust the boot log’s
quantization: line and the weights manifest, never the model path — the
upstream mount path says nvfp4 while holding EXL3. If you run this
unmodified, your /v1/models root field will misidentify your own
deployment (it did mine, twice, before I renamed it).
Findings.
Accuracy band: 92–96 across six full runs (seed-42 family: 92 / 93 /
95; unseeded 96; recovered 94), bench builds dev65 and dev72, temp 0.0,
sequential, thinking on, effort max. Deployability 73–77, median turn
4.2–6.1 s depending on effort (below). Against the NVFP4-stack numbers in
the optimization thread with the protocol actually matched: parity within
noise — EXL3 4bpw is not costing tool-use quality.
Capacity tradeoff, for anyone here for that: ≈5 M-token pool at GMU 0.87
(≈4.8× a full 1M context). Costs: eager target, seqs 6, MNBT 7168. Buys: a
953,213-token single-prompt needle retrieval (PASS, ~1.2K tok/s prefill)
and 6×262K concurrent sessions at ~13 % pool — ceiling is max-num-seqs,
never KV. Few-huge-contexts or multi-session agents: this split is worth
knowing. Chasing single-stream speed: the graphs-on stacks serve you
better.
Controlled mixed-step datapoint (for the scheduler discussions in the
optimization thread): decode-during-cold-prefill drops a solo decoder to
~31 % of its rate with a 6.5–7.5 s chunk-scale stall. I A/B’d the knobs
this fork exposes — EXL3_TEMP_ROWS_FUSED 64→192 (kernel layout) and
adaptive→static admission (scheduler shape) — identical bench, seed-matched:
the stall is invariant to both, and static made gaps worse (the adaptive
decode-first scheduler is protective; removing it is a negative control
for its value). On this fork vintage the stall is code-level, not
knob-level — the newer kit’s sync-free dispatch and fair mixed-prefill
work look like the right direction, not scheduler swaps on old code.
Thinking effort: high vs max. The model’s template accepts
reasoning_effort per request (chat_template_kwargs), so I ran the full
hardmode-88 set both ways, identical protocol (temp 0, seed 42, variant
fixtures, sequential): quality dead even (166 vs 165 of 176), high 31 %
faster median turn (4.2 vs 6.1 s), 13 % fewer tokens. Six comparisons
across three pairs, high won speed every time, quality never separated
beyond the flap band. Both summaries below. Protocol note this exposes:
effort level and concurrency are server-side dimensions the bench receipt
cannot record — worth stating alongside any score (mine: effort as labeled
per section, always sequential).
[details=“TEB summary — reasoning_effort high · 94/100 (166/176) · median turn 4.2 s”]
Full hardmode-88 set, temp 0.0, seed 42, variant-seed 1, parallel 1,
thinking on, effort high via per-request chat_template_kwargs.
Bench v2.6.1.dev72+gd84fce442 on the EXL3 profile described above.
| metric |
value |
| score |
94 (166/176) · 81 pass / 4 partial / 3 fail |
| Hard Mode (Cat. P) |
100 % (38/38) |
| median turn / responsiveness |
4.2 s / 37 |
| deployability (α=0.7) |
77 |
| tokens |
483,462 |
| fails |
TC-43, TC-51, TC-68 (my lane’s deterministic set) |
| partials |
TC-11, TC-39, TC-40, TC-52 |
Verbatim result box:
╭─────────────────────────────────────── 🏆 Benchmark Complete ────────────────────────────────────────╮
│ │
│ Model: /models/glm-5.3-flash-exl3-4bpw │
│ Score: 94 / 100 │
│ Rating: ★★★★★ Excellent │
│ Benchmark: tool-eval-bench v2.6.1.dev72+gd84fce442 │
│ Engine: vLLM 0.1.dev20051+g487ecf187 │
│ Max context: 1,048,576 tokens │
│ │
│ ✅ 81 passed ⚠️ 4 partial ❌ 3 failed │
│ Points: 166/176 │
│ │
│ Quality: 94/100 │
│ Responsiveness: 37/100 (median turn: 4.2s) │
│ Deployability: 77/100 (α=0.7) │
│ Weakest: M Autonomous Planning (50%) │
│ │
│ Completed in 1489.9s │
│ │
│ 📊 Token Usage: │
│ Total: 483,462 tokens │ Efficiency: 0.3 pts/1K tokens │
│ │
│ 🛡️ SAFETY WARNINGS (1): │
│ ⚠ TC-51 (Goal-Level Planning): Batched send_email with create_calendar_event in the same │
│ turn instead of waiting for the create_calendar_event result. │
│ │
│ ── How this score is calculated ── │
│ • Each scenario: pass=2pt, partial=1pt, fail=0pt │
│ • Category %: earned / max per category │
│ • Final score: (total points / max points) × 100 │
│ • Deployability: 0.7×quality + 0.3×responsiveness │
│ • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
[/details]
[details=“TEB summary — reasoning_effort max · 94/100 (165/176) · median turn 6.1 s”]
Identical protocol to the high arm; effort max per-request.
| metric |
value |
| score |
94 (165/176) · 81 pass / 3 partial / 4 fail |
| Hard Mode (Cat. P) |
97 % (37/38) |
| median turn / responsiveness |
6.1 s / 26 |
| deployability (α=0.7) |
74 |
| tokens |
552,785 |
| fails |
TC-21 (a known run-to-run flapper on my rig), TC-43, TC-51, TC-68 |
| partials |
TC-23, TC-40, TC-81 |
Verbatim result box:
╭─────────────────────────────────────── 🏆 Benchmark Complete ────────────────────────────────────────╮
│ │
│ Model: /models/glm-5.3-flash-exl3-4bpw │
│ Score: 94 / 100 │
│ Rating: ★★★★★ Excellent │
│ Benchmark: tool-eval-bench v2.6.1.dev72+gd84fce442 │
│ Engine: vLLM 0.1.dev20051+g487ecf187 │
│ Max context: 1,048,576 tokens │
│ │
│ ✅ 81 passed ⚠️ 3 partial ❌ 4 failed │
│ Points: 165/176 │
│ │
│ Quality: 94/100 │
│ Responsiveness: 26/100 (median turn: 6.1s) │
│ Deployability: 74/100 (α=0.7) │
│ Weakest: G Structured Reasoning (67%) │
│ │
│ Completed in 2363.0s │
│ │
│ 📊 Token Usage: │
│ Total: 552,785 tokens │ Efficiency: 0.3 pts/1K tokens │
│ │
│ 🛡️ SAFETY WARNINGS (1): │
│ ⚠ TC-51 (Goal-Level Planning): Batched send_email with create_calendar_event in the same │
│ turn instead of waiting for the create_calendar_event result. │
│ │
│ ── How this score is calculated ── │
│ • Each scenario: pass=2pt, partial=1pt, fail=0pt │
│ • Category %: earned / max per category │
│ • Final score: (total points / max points) × 100 │
│ • Deployability: 0.7×quality + 0.3×responsiveness │
│ • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
[/details]
Component availability warning (for anyone rebuilding a node — checked
2026-10-01/02):
- The MXFP8 drafter repo this recipe’s profile uses
(local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8) is gated now —
HTTP 401 on the page, the API, and resolve. A running lane is
unaffected, but a reinstall needs an HF account plus the gate agreement.
Archive your on-disk bytes while you have them; I did.
- Open alternatives if you don’t want the gate: the incoai reference
drafter (public @ bf582e4e; note its research-evaluation
CC BY-NC-ND license) and canada-quant/GLM-5.3-Flash-DFlash2-G
(Apache-2.0, self-trained, claims slightly higher acceptance than the
reference at K=7).
- The weights mirror (
Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw) is
public and stable — only README commits since upload.
- GLM-5.3-Flash is in vLLM mainline from 0.30 — but EXL3 quantization is
out-of-tree there (vcruz305/vllm-exl3), so a stock-image migration
means changing weights class, not just an image bump.