I know my timing is great (just when 3.8 Flash was released) but a bit over a month ago I asked myself a simple question: “Can we make Qwen3.5-122b-a10b faster while maintaining its quality?”. Albond’s provided me with a great hybrid model to use and it is still today that most of us still trust that model but I wanted a bit more.
So I went on a mission, but TL:DR First:
- You will need @eugr_nv’s repo to start with
- I only tested this in a GB10 chipset, should work on other hardware though
- You MUST apply the hybrid patch for proper performance
- The Hybrid Patch also works for @Albond’s old trusty Int4FP8 Hybrid model, so if you want to run the newest vLLM as of today (0.26.xx) with that model, you can use my patch
- I vibe-coded the heck out of everything :)
- If you follow the above, you should easily get 70+ tok/sec in coding.
- This model also got the Spark-AutoRound treament from @whpthomas
Why is it faster?
The model has 256 small specialist networks (“experts”) in every layer. It doesn’t use them all — for each word it generates, a router picks a handful and only those run.
I changed the router from picking 8 to picking 4. Nothing was deleted. All 256 experts are still in the file. The model just consults half as many per word, so each word costs about half the expert work which is where the speed comes from.
The catch: the model was trained expecting 8 opinions and now gets 4, so it loses a bit of quality.
That’s what the healing fixed, every layer also has one “shared expert” that always runs regardless of the router, and I retrained that one to make up for the 4 that no longer get consulted.
So basically, I told it to ask 4 specialists instead of 8, then trained its generalist to cover the difference.
In my (very custom) benchmarks, I recovered the entire quality that I lost originally:
Albond’s Hybrid Int4Fp8 - 51%
Standard INT4 AutoRound - 47%
This model TopK4-Int4 - 50.7%
(results are from 10 rounds average trimming the lowest and highest scores to remove outliers)
Also tool-calling should be on-par with what the base FP8 or INT4 models can do at around 90/100 on the tool-eval-bench benchmark (it would gravitate from 89 to 91 in my tests)
Please run it, try it, tell me the good, the bad and the ugly. I have the final checkpoint for about a week, I was doing quality analysis while in parallel improving the MTP head for better acceptance (spoiler alert, it barely improve things by about ~2%) so I’m happy with this checkpoint but I’m still working on getting it better if I find things I don’t like. So far so good :)
Get your model from here (launch parameters used in the development and testing are in the model card)
Don’t forget to patch your vLLM image!!! Otherwise you’ll be around 47-50 tok/s
