I got GLM-5.3-Flash running at 64 tok/s structured (62.6 at temp 1.0) on one NVIDIA DGX Spark.
Previous best published single-Spark numbers were in the mid-20s to low-30s. This also beats most dual-Spark DFlash2 results I’ve seen.
Full recipe + methodology + raw benches:
impressive, didn’t thought getting GLM-5.3 flash runnable on a single spark, was hoping an EXL3 version would pop up.
How is the actual performance / quality of the delivered code? Running qwen-3.8-flash-next now and it feels like a slow opus 4.5
i managed to run it, i would say that was the missing puzzle of the flash models, THANK YOU
be careful you can’t expect the artificial analysis agentic index score of 58 with a 2bits quant, expect 20-30% less and some unexpected behavior…
What’s the intelligence loss vs nvfp4?
isn’t the whole idea of 2bit EXL3 not that it is somewhat close to 4bit ?
2bit int quant, yeah that will degrade most likely
I tried it. It is not usable.
Several real world tasks got it to endless loops and hangs in parallel processing.
at the moment not stable enough.
BUT FAST. 70 tokens/sec when stuck in a loop. befor around 50.
I created around 1 million tokens with it in several benchmarks, too.
tested and no bueno quality is really poor. A for effort
First, make it work ✅
Then, refine and make it better ☐
| Quant | Size | Top-1 agreement | Mean KLD | Scored positions |
|---|---|---|---|---|
| Turboderp EXL3 2.05 | 85.23 GB | 88.92% | 0.121638 | 51,175 |
Full transparency - I was running 2bit v4 flash and was happy, same here - I’m happy. Its not perfect but I also don’t have tens of thousands of dollars to spend on a cluster than could run remotely close to a higher quant. I’m okay with the quality loss under the circumstances
I guess the question is, have you tried comparing running your work load on qwen3.8 27b or flash next on a single spark and seeing what handles your tasks with less issue?
What are you using it for?
I tried with real task. The server kept crashing. I can attach log, if it’s interesting.
Correct question. Azempatti’s 1-sparks Qwen 3.8 Flash next runs at 45-70 t/s and quality is absolutely fine. 90+ on hardmode TEB.
Don’t kid yourself. What’s the point of all that speed under this quantization setup? Speed only holds value when it’s built on top of model capability.
I was able to successfully deploy it with changes suggested in PRs 3 and 4. But the max speed is 50 tok/s before PRs (it crashes after ~10ktokens) 40 with PRs applied (works reliably). Not bad still, but please dont advertise 60 tok/s.
again its really apprechiated effort from you, but it get crach after sometime, is there a fix for this?
