I’ve tried Laguna, Qwen 3.6 35b, DS4 flash, and a whole host of other models (I keep a running benchmark database of speeds and recipes and setup like I’m sure many of you do).
I keep coming back to bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid · Hugging Face as my daily driver. What are your experiences? I’m currently tuning it for agentic tasks for Hermes and Openclaw (with some learnings from the latest Laguna recipes) and i’ll post my recipe here. I think we can beat what’s currently posted in terms of recipe for maximum throughput.
Same experience. I’d go on to say it’s the best balanced model on my two spark cluster. Smart, good tool handling, reasonable speed. I find Qwen 397b, M2.7 and DeepSeek 4F to be just beyond tolerable in token generation speed. Love the speed and tool-calling of Qwen 3.6 35b-a3b, but it is missing too much world knowledge.
Feels like there are very few new “middle-sized” options in the ~100B parameter range. Was hopeful about Laguna, but it looks very slow.
That’s the tradeoff of the 35b isn’t it. General knowledge vs. speed. And the tradeoff on code is 3.6 35b vs 3.5 122b almost the same from the generational difference. Seems like there’s a perfect spot for a 60-80b Qwen 3.8.
Just curious what kind of speed you are getting from 122b on 2 sparks if ds4f is just tolerable? Fp8 122b is about 35 t/s at best times, mtp is multi pass very old style, run on marlin or deepgemm no chance of flashinfer. I registered no tool call quality improvement over int4, most likely due to use of suboptimal kernels. Meanwhile ds4f DSpark does 45-50 t/s on single, up to 240 t/s on 16 reqs and provenly works coherent on up to 1m tokens. 122b is probably still best large model for 1 spark but two? You must be cooking wrong
It’s fair, I was thinking about fp8, int4 should definitely be faster on 2 nodes, but 65 meaning you run a well optimized stack, on more less vanilla I haven’t seen more than 40-45 from the memory.
My experience has been mostly the same. Qwen 3.5 122B just seems like the simplest choice to have the largest actually capable model in memory at a pretty high context. But a lot of people on here have been optimizing DSV4F, so I’m kind of hopeful. I think that maybe at some point, someone could get a higher quant of it running than what is used in antirez’s repo, and even at lower speeds I think it would beat out 122B… at least for LLM use. 122B also has the advantage of having a pretty capable image model, and combined with its intelligence, it makes it absolutely excellent for image interpretation (especially translation) tasks.
What I’m kind of curious about is whether someone could do more work with 122B that doesn’t involve the Autoround quantization. IME this intel 4 bit autoround quant still has a very real tendency to loop, and I’m not sure whether it’s actually the best in terms of fidelity. For instance, maybe the NVFP4 variants of 122B could actually perform better in terms of just output quality? Just food for though, not sure if 122B has even been fully optimized on 1 spark. The official Nvidia repo version of this loops because they apparently screwed up the quantization themselves.
Laguna is still very new, so room for improvement but… my results haven’t been good.
I just did and posted it to my github - massively increases KV cache and overhead optimization on the qwen 122b hybrid model by blesyg, so you get maximum intelligence, speed, context and KV cache. I had to do a lot of patching of the unpublished latest VLLM v26 build:
I think at this point, this is the absolute maximum squeezing we can do of qwen 3.5 122b. 5 lanes at 256kb at 40+ t/s decode is pretty phenomenal.
So I was running qwen3-122B on a dual node cluster, but for me, it kept making so many mistakes like making assumptions and stating incorrect data, it even got my name wrong and called me Jean?! I have hindsight memory setup and first thought it was my agents’ setup. Yesterday I changed to deepseek-v4-flash (FP8 MoE) and I’ve got so much further in my coding and reasoning. maybe it’s just subjective??
Deepseek4 with dspark on 2x sparks is a game changer. It feels the same as sonnet to me but just a smidge slower. That’s honestly all you could ask for with prosumer grade hardware.
I’ve been rolling with the new Laguna S-2.1 single node recipe and it’s been rocksolid at 45 tk/s with great follow through and effective tool orchestration
IMHO, Deepseek V4 Flash is the best daily drivers. I have tried most popular 122b, 27b, 35b A3b and the Deepseek V4 Flash slightly better in development processes then 27b - and this is the great resuilt. As for the agents you need additional blasting fast model for dumb subagents (for ex., Qwen3.6 35B A3B NVFP4) to do some garbage work and not to touch the main reasoning model.