I too shall take a look
Hi tenari,
Thank you very much for your work! I did not have time to test your previous models. This one really attracted my attention due to faithfulness score.
I just completed my tests and would like to share with you. In short, the results are amazing both in terms of speed and accuracy - simply better!
Loads & serves cleanly. vLLM picks up the mixed-precision menu correctly — CutlassFP8ScaledMMLinearKernel (FP8 rung) + FlashInferCutlassNvFp4LinearKernel (NVFP4 rung) — weights load in 22.3 GiB / 13.6 s. MTP-3 (--speculative-config '{"method":"mtp","num_speculative_tokens":3}') coexists with --enable-prefix-caching with no drafter hang. One note for FlashInfer users: with spec-decode, cudagraph auto-downgrades FULL_AND_PIECEWISE → PIECEWISE (expected).
Throughput (N=1/2/4/8 concurrent, 250 output tok, no-think, ignore_eos):
| N | PrismaAURA NVFP4 + MTP-3 (tok/s) | our prior FP8 + MTP-3 |
|---|---|---|
| 1 | 19.0 | 14.5 (+31%) |
| 2 | 38.4 | — |
| 4 | 74.2 | 66 |
| 8 | 122.6 | 116.6 |
MTP-3 acceptance: mean accept length 2.80, per-position 84 / 56 / 40%. Interesting detail vs our FP8 build (3.09, 86/67/56%): AURA’s draft head accepts a touch less, yet AURA is faster single-stream — the smaller NVFP4 weights make each forward cheaper on this bandwidth-bound box, which more than offsets the lower acceptance.
Faithfulness — grounding (Vectara HHEM-2.1 cross-encoder over contract-extraction prompts): grounded answers mean 0.834 (7/7 faithful); on 3 trap prompts (answer absent from source) it abstained 3/3 with no fabrication. The quant did not degrade grounding on our extraction workload.
Faithfulness — BF16 divergence: AURA greedy tracks BF16 greedy for 18 tokens on average (median 15, max 54 of 64) before first divergence — about the same closeness as FP8 tracks AURA (19 tok).
Speed vs full precision (N=1 single-stream): AURA 19.0 tok/s vs BF16 4.8 tok/s = ~4× faster, vs FP8 17.0 = +12%.
Grounding (HHEM) identical across AURA / FP8 / BF16: 0.834 / 0.826 / 0.826, all 7/7 grounded + 3/3 abstain — the NVFP4+FP8 quant cost zero grounding quality (AURA marginally above BF16).
Thanks again!
Alper
Wow!!! I’m hoping 3.7 open weights drops soon so that I have some new models to play with.
I don’t have enough hardware for GLM-5.2
Hi Rob,
Your AURA quant is excellent — been running it on my Asus GX10 since yesterday. Did a serving-config sweep with different recipes i saved before. (single Spark, GB10, N=1, kept it conservative since I had another job on the box) and got the best throughput serving it on the AEON vLLM image: holds 22+ tok/s decode out to 100k context, loads clean, snappy on repeated long context, no crashes. Detailed report is attached. All recipes, llamabenchy numbers are in the report. If you see any thing to improve let me know. Open to suggestions.
Separately — if your MiniMax M2.7 quant is finished, I’m happy to test it on the AsusGX10.
Thanks for your contributions.
prismaaura-spark-serving-report.md.pdf (658.3 KB)
Awesome job but I have a question, there is a reason for --max-model-len 32768 in model example recipe? for my usage full context length of Qwen3.6 is a sweet point.
Another awesome quant.
Been running it since last night and so far it seems even better than prismascout, which was already my best 27b quant. Even at 120k context I can’t really tell the accuracy difference between this prismaaura and fp8 as a coding subagent. Amazing job!
Thank you for your efforts! It works great on my real-world tasks!
I created a 4.75 prismaquant version of Ornith-1.0-35B: cyburn/Ornith-1.0-35B-4.75Bits · Hugging Face
I had to fix the current Prismaquant code to make it work… but CLaude code helped me do it.
Hello, I try to use prismascout quant with opencode and often it breaks my session when it’s needed to use write file, command tools or even on delegating job to subagents. At vllm log I see that generation speed becomes 0, and opencode still reports “writing command”, “delegating…”. I use --tool-call-parser qwen3_coder.
Did anyone face the same?
I use only qwen3_xml on my machine because from the tests I ran with qwen models it gives me better results compared to qwen3_coder.
I’m going to try to ship ornith 35B this weekend :)
I should have shared the mods I had to make to the code to quantize it to 4.75 bits last weekend.
Okey… so I have been heavily working on a benchmark that should be able to identify differences in models’ intelligence and capabilities (we already have a great Tool-Bench one, so I’m focusing on what’s important for my workloads, maybe one day It’ll be more generic so I can be able to share it here.
I built it so Sonnet would get ~80% and Opus 4.8 get high 90s, but at the same time, I can run the same model with NVFP4, INT4 and FP8 and get different results based on their quantization.
While this is a very Early Beta, I can already see the trends: for instance, Qwen3.6-35b NVFP4 being very inconsistent compared to the custom AutoRound-OC version that I made with @whpthomas’s method. Also Albond’s hybrid Qwen3.5-122b scoring more than122b-int4 while also being as consistent, and when running 35b-a3b-FP8, I’m getting an EVEN more consistent scoring but sometimes less ‘intelligent’ than 122b for obvious reasons).
Now, l just ran Ornith-PrismaAura once (still pending the consistency test) but the intelligence one just smoked both Qwen’s 35b and 27b by almost 15% and 122b by 10%. This is the first Local model that I can get to score higher than 60% in this benchmark.
I’ll keep doing analysis and then validate the logs for accuracy and report back, but this very first bench gave me the excitement to try this model for this coming week in production :)
I have been running a lot of sensitivity analysis on different settings – trying to extract the highest quality out of these smaller faster models. This latest one examines the role of temperature on tool eval scores. The results are jagged, which is perplexing. Not sure what to make of it. No clear pattern.
Like fp8-t0.4 is has the highest and sar-t0.4 has the lowest score ¯\_(ツ)_/¯
A temp of 0.7 seems like the most consistent, at least between these two models.
TEMP — Tool Eval Benchmark Results
Ranking
| # | Model | Quant | Score | Min | Max | t/s | P/F | Weakest | Runs |
|---|---|---|---|---|---|---|---|---|---|
| 🥇 | qwen3.6-35b-fp8-t0.4 | FP8 | 94.00 | 94 | 94 | 48 | 183/24/0 | 🟡 Restraint 83.00% | 3 |
| 🥈 | qwen3.6-35b-fp8-t0.7 | FP8 | 93.00 | 93 | 93 | 48 | 180/27/0 | 🟡 Restraint 83.00% | 3 |
| 🥉 | qwen3.6-35b-fp8-t0.2 | FP8 | 93.00 | 93 | 93 | 48 | 180/27/0 | 🟡 Restraint 83.00% | 3 |
| 4 | qwen3.6-35b-fp8-t0.5 | FP8 | 92.00 | 92 | 92 | 48 | 177/27/3 | 🟡 Restraint 83.00% | 3 |
| 5 | qwen3.6-35b-sar-t0.7 | INT4-AutoRound | 91.00 | 90 | 92 | 65 | 175/27/5 | 🟡 Instruct 80.00% | 3 |
| 6 | qwen3.6-35b-fp8-t0.1 | FP8 | 91.00 | 91 | 91 | 48 | 177/24/6 | 🟡 Restraint 83.00% | 3 |
| 7 | qwen3.6-35b-fp8-t0.3 | FP8 | 91.00 | 91 | 91 | 48 | 177/24/6 | 🟡 Restraint 83.00% | 3 |
| 8 | qwen3.6-35b-fp8-t0.6 | FP8 | 91.00 | 91 | 91 | 48 | 174/30/3 | 🟡 Context 80.00% | 3 |
| 9 | qwen3.6-35b-fp8-t0.9 | FP8 | 91.00 | 91 | 91 | 48 | 174/30/3 | 🟡 Safety 81.00% | 3 |
| 10 | qwen3.6-35b-fp8-t0.8 | FP8 | 91.00 | 91 | 91 | 48 | 177/24/6 | 🟡 Context 80.00% | 3 |
| 11 | qwen3.6-35b-sar-t0.5 | INT4-AutoRound | 90.67 | 88 | 93 | 65 | 174/26/7 | 🟡 Code 72.00% | 3 |
| 12 | qwen3.6-35b-sar-t0.8 | INT4-AutoRound | 90.67 | 88 | 93 | 65 | 173/29/5 | 🟡 Code 72.00% | 3 |
| 13 | qwen3.6-35b-sar-t0.9 | INT4-AutoRound | 90.33 | 89 | 91 | 65 | 174/26/7 | 🟡 Instruct 73.33% | 3 |
| 14 | qwen3.6-35b-sar-t0.6 | INT4-AutoRound | 90.33 | 90 | 91 | 65 | 170/33/4 | 🟡 Code 72.00% | 3 |
| 15 | qwen3.6-35b-sar-t0.1 | INT4-AutoRound | 90.00 | 88 | 93 | 65 | 174/25/8 | 🟡 Code 72.00% | 3 |
| 16 | qwen3.6-35b-sar-t0.2 | INT4-AutoRound | 90.00 | 87 | 92 | 65 | 171/30/6 | 🟡 Create 72.00% | 3 |
| 17 | qwen3.6-35b-sar-t0.3 | INT4-AutoRound | 89.67 | 88 | 91 | 65 | 171/30/6 | 🟡 Create 72.00% | 3 |
| 18 | qwen3.6-35b-sar-t0.4 | INT4-AutoRound | 89.00 | 89 | 89 | 66 | 170/29/8 | 🟠 Plan 67.00% | 3 |
I’m running everything on t0.0 and with thinking disabled. Completely different approach with lot’s of math in play.
I found that trying to be deterministic when asking for coding tasks (which is what I use 99.99% of the time) it’s VERY hard to benchmark properly. So I’m trying to see what other capabilities of a model can be squeeze out from them when pushing a model to its limits.
I’m already at a position to be able to guess only by the numbers I get as result, the Quant Type of a model (i.e.: Int4 vs FP8) very easily. But I’m halfway there to be able to quickly identify (for example) 122b-a10b-int4 vs 35b-a3b-fp8, They are still very close on results, what changes is repeatability on higher quants, and accuracy on higher parameters models.
If I get it to be more precise on the tasks, I might share it, but for now, comparing all 35B models and quants it’s awesome with this process. I’ll show some example results tonight/tomorrow (I’ve updated the bench so I’m re-running everything now, which runs 10x per model so it takes a while)
Unfortunately we need a bit of the stochastic sugar for production – I think – so I am trying to be representative of the target workload. Do you run t0.0 and with thinking disabled in production?
Yes and no. t0.6 mostly but t0.0 on some debugging runs (it’s like a 90/10% split).
Can you share your recipe?
To keep it on topic, I’ll share the Ornith-PrismaAura , but I basically copy-paste my Spark AutoRound one for this one ;-)
cluster_only: false recipe_version: '1' solo_only: true defaults: max_num_batched_tokens: 32768 max_model_len: 200K host: 0.0.0.0 num_speculative_tokens: 3 gpu_memory_utilization: 0.50 max_num_seqs: 8 port: 8000 served_model_name: rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP env: VLLM_MARLIN_USE_ATOMIC_ADD: '1' VLLM_HTTP_TIMEOUT_KEEP_ALIVE: '600' VLLM_ALLOW_LONG_MAX_MODEL_LEN: '1' model: rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP description: rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP with native MTP speculation, FlashInfer attention, long-context Spark profile name: rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP container: vllm-node-tf5-20260623 command: | vllm serve rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP \ --host {host} \ --port {port} \ --served-model-name {served_model_name} \ --max-model-len {max_model_len} \ --max-num-batched-tokens {max_num_batched_tokens} \ --max-num-seqs {max_num_seqs} \ --gpu-memory-utilization {gpu_memory_utilization} \ --dtype bfloat16 \ --moe_backend auto \ --load-format instanttensor \ --attention-backend flashinfer \ --enable-chunked-prefill \ --trust-remote-code \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --speculative-config '{{"method":"mtp","num_speculative_tokens":{num_speculative_tokens}}}' \ --override-generation-config '{{"temperature":0.6,"top_p":0.95,"top_k":-1,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}}'
vllm version is 0.23.1rc1 , I keep the name with the date I built it :)