Hi everyone,
A few people have given me feedback, and I also wanted to evaluate the quantization more objectively myself. Based on those evaluation results, I made some further adjustments to the layers selected for quantization and ran another round of quantization.
The result is a second quantized version, still using the same MixedInt4 approach.
| Evaluation | BF16 Original | MixedInt4 | Difference | Recovery Rate vs. Original |
|---|---|---|---|---|
| MMLU | 0.8349 | 0.8307 | -0.0042 (-0.42%p) | 99.50% |
| GSM8K flexible-extract | 0.7286 | 0.7612 | +0.0326 (+3.26%p) | 104.47% |
| GSM8K strict-match | 0.7036 | 0.7369 | +0.0333 (+3.33%p) | 104.73% |
As you can see, the recovery rates are very good, and I think this second version turned out to be a very successful quantization.
I am also planning to run Tool Eval Bench on this version and post the results in a new reply comment or thread. The updated model itself will be re-uploaded to the existing model repository.
I am currently studying at an academy while preparing for employment, and I am on my way home now. I will probably be able to start testing the community benchmarks in about an hour.
Thank you to everyone who has tried the model and provided feedback.
I believe the experience I gained from this process will help me provide an even better quantized version when the Qwen3.8 35B MoE model is released.