I’ve tried minimax m2.7 4bit AWQ and Qwen 3.6 27b, Qwen 3.5 122b, Qwen 3.6 35b, pretty much every opus and sonnet distill of the Qwen models, the new gemma models and while their pretty good for things like coding, their real world knowledge is always lacking which yes very expected at 27b and 122b parameters. Anyone know of a good model? I’m at a loss at this point and i’d really like to reduce my reliance on Claude. I have tried deepseek v4 API and its decent but its honestly felt worse than minimax m2.7 AWQ 4bit.
Have you tried Qwen 3.5 397b?
Small, local models should be given a means for web searching. Do you have a method in place?
I agree with @jrsphd. It makes small models like Qwen 35b more useful. Try using SearXNG. It might help with your purposes.
Thank you for your reply! Yes I have tried local web search and while it does help a lot most models will get confused by search results and mix info leading to a incorrect answer. This is running the smaller models at fp8 as well.
Hmm, the closest model that is similar is Qwen3.5 397b, but even that is just a ~400b model. Have you tried REAPs of the larger models yet? You can even create your own, but you’d need to calibrate by renting (to run the larger models).
Most REAP models are GGUF and I thought that vllm doesn’t really support gguf all that well.
Honestly, it’s going to be tough to replace Opus or GPT (which are 5–10T parameter models) on a local rig. The closest open-source giant we have is DeepSeek4 at 1.6T parameters. The best open model for general knowledge is Qwen 397B (for dual spark setup), but it’s still 10 to 25 times smaller than those proprietary cloud models…
GGUF isn’t the only format for quantized models. But the real issue is that quantized models perform almost as well as the full ones in coding, while you’re looking for broad general knowledge… (though I could be wrong).