You probably saw Alex’s video here: https://www.youtube.com/watch?v=zBYfzecY5ww
Well, I wanted to run this benchmark on my “usual suspects” models and see what story this benchmark had to tell, there were surprises and some expected results. If you’re not familiar with this bench, it has by default two codes to run it against (you can supply your own!): http_server (shorter, friendlier for lower context models) and JQuery.js (10k lines) for big boys:
The results are only for the JQuery one, the http_server was a bit more “relaxed” in terms of thoroughness but still there were differences. JQuery makes them really obvious.
Finally, there’s a “–relax-indent” parameter that I HAD to use as it’s a known issue that some models will reply with identical data but different indentation which the benchmark recognizes as incorrect. Worth mentioning Qwen3.5-122b-a10b-hybrid scored the same with and without this parameter.
Don’t just look at the bar and lines on the above result, look at how many lines did Q3CN hallucinated! And this is the one bench that answered my question on WHY I couldn’t rely on this model. I tried and tested a million times, it would create really cool stuff, but when adding/fixing code I struggled with it. Now I know why, it’s very creative, too much! (of course temperature is set to 0.0 by the bench and also it was at 0.2 from my recipe, so that should be fine).
The 35b-a3b-nvfp4 always does 16 lines (PSEUDO Function), and sometimes there is a few more, out of 9 runs (yes, nine), It showed 20 hallucinated lines twice hence why I kept this result.
122b? - Solid! Also… watch Alex’s video and compare the results with the Anthropic model he tested, 122b wins over that in the same benchmark. Crazy!
The above shows a bar per functions, that’s very relatable with the one below, read on :)
The last one is hard to see because they are all very similar, the not is that 122b NEVER dipped, also both 122b and Q3CN-FP8 where able to solve the third scenario (PSEUDO in the bar chart), but Q3CN crashed and burned at a high length position.
All 35b-a3b variants (all of them) failed the Pseudo scenario hallucinating 16 lines. Everything else was ON PAR with 122b. Impressive.
I purposely hidden the 35b-a3b-FP8 version since I had the wrong chat template and it didn’t go well :) That was user error,
Also, there are two NVFP4, different recipes but within per-run variance. Just kept them both to have like a min-max for that model.
Overall… I’m happy that I can see on a what I can perceive using my daily workload, this + Introducing Tool Eval Bench CLI are my two workhorses to evaluate models. If tool-eval tells me a give model handles tools well, then that relates to a 1-1 my Claude Code won’t crash/hang (anything above 80-82 score works well, above 90 is rock solid to me).
And Codeneedle, now it tells me also very accurately if I will like the coding workflow that I use my models to. Check out his repo here GitHub - alexziskind1/codeneedle · GitHub
Just wanted to share this with y’all!
EDIT : Updated the first graph to include Qwen3.6-27b-Int4-Autoround., in that one looks the same as 35b-a3b (failing Pseudo as well), but did just a TINY BIT better on the recall-reposition one, I’ll upload that tomorrow.