Benchmarked a few models on 2x Sparks

Started off wondering which model of glm 5.3 flash to run on 2 sparks, got distracted and tested a few other models…

Not meant to be most statistically accurate, but thought I’d share.

Used for all models:

  • tool-eval-bench 2.6.1.dev66+g32862e97a
  • temp 1.0 and top_p 0.95
  • seed 42
  • the average of the score and time taken is used for multiple runs
  • parallel 1

A few observations:

  • all models are, on average, in the excellent category, so any of them will give excellent results.
  • perhaps tool-eval-bench is not the best basis to judge for all use cases - there are too few test, and the tests are too easy. Most models are competing to get only 1 or 2 extra points. However it does give a rough indication. People testing with 1 run until get max score are overestimating, many of these got a 93 or above at least once.
  • Qwen 27B dominates the frontier for this test. FP8 is amazing quality, and nvfp4 is amazing speed with decent quality

Note:

  • I may not have latest setups, which may shift speed a few percentage points to the left…
  • change in temp and top_p could completely change this, but this is how I run them in production.
  • need to run many more times for each model, but I’ve burnt enough time, need to get back to using them for work…

While I love 27B, for my use case I feel like GLM 5.3 flash gets to the point quicker and more accurately, and feel I can trust leaving it alone a bit more.
I use low in general use or max when want detailed plan or overnight usage. I also swap in ds4-vision about 40% of the time.

Same impression for long running independent tasks. GLM 5.3 “gets it” fast and continues by itself even where you haven’t even explored. While 3.8-27b is fast to get it “done what you give” not a dime more. My workflow is in the first camp and my problems are better fitted to be solved by GLM 5.3. Its slower, but I take that slowness and more correctness and independence over fastness and babysitting of 3.8-27b.

Updated with Technigmaai recipe, and man it is far superior to other GLM5.3 flash recipes. Much faster, and very smart on max.

Basically dominates every other GLM5.3flash recipe that I have tested so far.

Also added mimo2.6 to graph and greyed out recipes that have a better alternative for that model (either faster or smarter).

Have been running Technigma recipe for a few days now and it gives me the good feeling that I can rely on it. Also fast enough for conversation and interactive work.

Which Technigma GLM 5.3F recipe did you use? He has two posted: GitHub - technigmaai/glm-5.3-flash-nvfp4-2x-dgx-sparks: Qualified GLM-5.3-Flash NVFP4 deployment for two ARM64 NVIDIA DGX Spark nodes · GitHub and GitHub - technigmaai/glm-5.3-flash-gb10-tp2-native: GLM-5.3-Flash NVFP4 on two NVIDIA GB10 nodes: native TP2, one container per node, display-backed KV and image input · GitHub . I’ve seen someone else benchmark the latter which suggests it may even be better (I’ve been using it with good results so far).

Its the latter one, I wasn’t aware there was another. Very happy with this one.

How many runs did you do? These seem to be inconsistent depending on how many runs you do, especially at high temp.

I’ve encountered some looping and lower output integrity on the only NVFP4 GLM5.3F variant I have tried, so I have been sticking with EXL3. Been considering trying that particular NVFP4 repo, but need to clear up space.

I’ve found Tensorfold to be… weird.

Anyway my task most of the time isn’t TEB anyway, so it’s kind of questionable usefulness to me.

And yeah the point of GLM5.3F vs say 27B is that it’s a more strategic thinker, so you can leave it alone. 27B just churns out code great though. My plan is to eventually clear up time to get my 5090 running 3.8 27B and then run 5.3F at the same time on my 2x Sparks and have the former produce code very quickly while the latter handles planning for the task and code verification. Just so drained from work lately.

I tried ninfer qwen 3.8-27b on my 5090. its damn fast (~130-140tok/s on coding) on NVFP4. But, and big but there is ninfer has some chat template problem that does not quite work well with Hermes from quite often. after every 4-6 messages, get that weird error that GLM retries but can’t get it working. I have to type up some random message/change thinking level back and forth and that thing again starts working.

You may try Tensorfold 27b. I only have it on Mac M5 max w upto150tg, consistent 93 on hardmode, . Not sure on CUDA side

The number of runs are after each observation on the graph. Agreed, I try to run at least a few then take mean so as not to get a random outlier. e.g. (7) for glm53f max technigma-nvidia.

What I am trying to understand is that some recipes result in large variance (mia 92-95) and others have zero variance (like mimo, technigma). I would have expected at least some variance at temp 1.0. But maybe I just need to do more runs.

Oh, sorry, I missed where the number in parentheses was noted then. I’m wondering if it’s some sort of chat template issue, since I feel like most of these should perform similarly enough.

I asked GLM itself about quantization types earlier, fwiw, and it said that the EXL3 variants should on average retain the most broad information, while NVFP4 is more focused? My layman interpretation. I wouldn’t be surprised if the well put together NVFP4 variants just consistently did better at coding, though having 7 scores of 95 in a row at temp 1 is weird.

For my current task though, that benchmark is only of limited use, I mainly need general knowledge/intelligence and long context coherence/memory. It’s not doing any coding.

Thanks, always looking forward to recipes since I haven’t had time to set up that box yet.

Updated:

  • Fixed Frontier line
  • added TP3+TP4
  • added deepseek 1017
  • removed dominated items

Currently deciding whether better to run ds1017 on 3/4 nodes, or glm53f on 2 with ds4-vision or something else on other 2 nodes. Would be nice to have second opinions on coding. However with Flare, I am able to start up a model in less than 30 seconds, so can also jump between models that way