Collecting eval results for Spark-sized quants of models

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Until there’s something better, I decided to start running and recording some basic evals on the models I can run locally so I can more easily compare them.

I’m starting with arc_challenge and mbpp. I wanted evals I can run completely within a container, although I might change my mind on this given how many newer benchmarks require the ability to spawn containers. I also wanted something where I can use a single framework (Inspect AI in this case) to run different evals without having to figure out each one individually.

My goal here is to be able to answer questions like:

  • Is Gemma4 26b better or worse than Qwen3.5 35B?
  • What impact does using --quantization fp4 --kv-cache-dtype fp8 have on accuracy?
  • Is Qwen3-Coder-Next better than Qwen3.5-X?
  • Exactly what command/flags are required to get those specific scores I recorded?
  • Is this AutoRound better than this NVFP4?
  • Is this fine-tune from an anonymous egg avatar actually better or worse?

I’m recording my results (including the instructions/commands I used to collect them, and for every tested model, the commands and flags to run that model) here:

Ultimately it’d be nice for this data to live somewhere like spark-arena (and it sounds like that might be in the works?), but in the meantime I wanted to start collecting some numbers for my own comparisons.

Note: My plan is not to run all benchmarks for all models, just get a small but broad sample that can be used to get an approximate idea of the relative accuracy between models. I’m not concerned with speed at all here (those metrics are available on spark-arena).

Feedback (and other results) welcome!

2

Hugging Face introduced a feature for storing community evals.

A wider audience. And possible more contributors than us “few” Sparkies.

3

I’d just like to highlight that you’re taking on a very ambitious task.It’s challenging already because the utility and behavior of models change depending on the agent version or updates to the inference systems (bugs get fixed, etc.).

For example, when the 3.5 models were just released, they were worse at writing code than Qwen3-Coder-Next. Today, the 122B model writes code significantly better than it did at launch — and better than Qwen3-Coder-Next does now.

There are just so many changing parameters. It seems to me that if you freeze the software versions, the results won’t be that useful.

Honestly, it’s great that you want to create a series of test cases. But in the end, they might only show improvement or degradation «under changed circumstances», and you won’t be able to definitively say something like «Qwen is always better than Gemma» and so on.

It also seems that different models may perform better or worse depending on the specific task.

So, ultimately, it’s up to you to decide for yourself: which model is better for you — Gemma or something else.

(It’s a bit frustrating that Qwen3-Coder-Next was solving one particular task quite well just a couple of weeks ago (in 11–15 minutes), but today I can’t find any combination of inference system version, agent, and parameters to reproduce that result. It can’t solve the task even in 30 or 60 minutes.)

4

Oh, I didn’t know about this. It does look very useful, although it doesn’t look like it handles differences in flags passed to the runtime (for example I want to see what difference --kv-cache-dtype fp8 makes).

5

I’m not sure it’s that ambituous. I’m not planning on doing all benchmarks for all models, I’m just just putting the numbers I’m already collecting (to get a rough idea of how they compare) somewhere central so I can more easily refer back to them when a new model comes out.

I also often wonder if a flag like --kv-cache-dtype fp8 is a good choice. It’s hard to know what difference this makes to accuracy.. But if I run with/without on the same model in the same week using the same versions of everything, it might give a reasonable indication of whether the accuracy loss is small enough to be worthwhile.

Yeah, I had thought about this, and I think the fix for this is to re-run them if it seems like the current results might be outdated (for example Gemma4 got a lot of vllm fixes after launch). My intention is not to build a super-accurate database of all scores, just to provide some approximate numbers to allow a rough comparison between models. The published unquantized scores are not useful if you have to use different sized quants of each to fit on Spark.

Yep, I agree. I will always use the latest version of everything when I run them. The results have all the timestamps on them (so it’s possible to infer which versions of things were run on), and if there are significant fixes made to models, they can be re-run.

Yeah, this is my next question - what combination of these gives the best results. But first I want to narrow down the selection of models, so I can then compare those 😄

6

I like the Idea. How can I contribute? Especially interested in Gemma 4 31B and Qwen3.5 122B Int4 Autoround. I have a spark.

7

Instructions to run the evals in a container are in the readme (they’re written assuming you already have the model running, you just point the evals at the endpoint):

I suspect the Gemma4 31B model will take a while (I think it was around 8-9 hours for the 26B on mine, and the dense model will be many times slower), but it would be interesting to see the comparison.

If you do record any and send a PR, please be sure to include the exact command you use to run the model (some flags may affect accuracy and we should capture that) in the readme (you can see the existing scores folders as an example).

Thanks!

8

I’ve decided that only running the few benchmarks that work inside of a container is too limiting - the scores are high even on the small models and I think the more interesting benchmarks are agentic tests (like AgentBench) that will need to spawn their own sandboxes/containers (and therefore don’t work inside another container).

So, I’m going to start running the evals from a VM (on another machine using multipass, so things are still isolated, but docker is available). I’ll probably update the readme to reflect that later this week. This does mean it’ll be a bit more involved in running them, but it’ll should provide more useful numbers.

9

Could you expand on this? I don’t think the weights have changed substantially so what has changed to make Qwen3.5 better on coding today?

Harness evolution? Hermes? Claude Code leak? Genuinely curious.

10

You may already be aware, but it’s possible to map the docker socket into a container so that it can manage containers on the host. Not great for security, but if you control that particular image, reasonably safe.

11

Thanks - I am aware. I do sometimes use it (such as for my dashboard where I control the code). But in this case I’m not familiar enough with all the code/deps running with these benchmarks to use it. Perhaps I should look into it… if --sandbox docker ensures all third party code runs inside containers (which don’t get the socket) and the only code running outside is InspectAIs, it might be safe.

But then, the multipass VMs turn out to be pretty easy too (it’s fairly similar to docker CLI, but for VMs) so it might not be worth it.

12

I dropped some of the benchmarks where all models were scoring high, and right now I’m just running AgentBench and IfEvalCode (ts/c#/shell).

I’ve been running only on the first 300 samples for each for now (in an attempt to get through more results), and on a dedicated machine (to avoid any resource issues in the VM), but I’m not sold on the usefulness of these results as-is.

If I run multiple times, the scores vary wildly. I think you’d need to run the entire set for many epochs to get a reflective average.

Everything getting 0% on TypeScript looks suspicious, and the time differences are also massive. The FP8 version of Qwen3.6 somehow scored higher than the full size model, and did it in way less time.

Screenshot 2026-04-19 at 09.43.26
Screenshot 2026-04-19 at 09.43.261214×1280 193 KB

I think I’m gonna have to start reviewing the logs to understand the failures. I’d hoped it would just be as simple as running them and collecting numbers 🙃

Full results are in the readme at GitHub - DanTup/spark-evals: Some benchmark results of small models and quants that fit on DGX Spark · GitHub

13

Greetings,

I’d like to share some thoughts regarding KV cache quantization:

  1. Architecture matters. To understand if KV cache quantization is worth it, we first need to look at the model’s architecture. For instance, we have models with classic Attention (Qwen2/3), Hybrid Attention (Qwen3.5 / Qwen3-Next), and Linear/Mamba-based architectures (Nemotron-3).
    Classic attention generates a massive KV cache that significantly impacts generation speed. On a performance graph, a model like Minimax with classic attention would show a much steeper decline as the context grows.

Screenshot_20260421_000328
Screenshot_20260421_0003282358×1244 212 KB

  1. Active parameters vs. Performance. Performance also heavily depends on the number of active parameters. In models like A3B, the weights are very small, and even Hybrid Attention cannot fully prevent performance degradation. However, the difference between classic and hybrid attention remains significant.

Screenshot_20260421_000037
Screenshot_20260421_0000371809×1153 149 KB

  1. The case for Linear Attention (Nemotron). I haven’t tested Nemotron yet, but since it uses linear attention, it should theoretically perform better than Qwen 3.5. This raises a question: is there any point in quantizing the cache for such models? We likely won’t see a speed boost. It might help fit more context into VRAM, but given that linear attention cache is already compact, is it even necessary?

I apologize for theorizing without hands-on verification—I will definitely run some benchmarks on these models in the near future to confirm this.

My main takeaway is this: we should only consider KV cache quantization if it provides a significant speed boost (e.g., models with classic attention or hybrid models with fewer than 3B active parameters) or if VRAM is extremely limited.

For instance, with a massive model like Qwen 397B, there is almost no memory left for the cache; in this case, switching to 8-bit KV cache is a necessity just to squeeze the model and the cache into two Spark nodes. In most other scenarios, quantization seems unnecessary. It may degrade model quality without offering any meaningful advantages.

Additionally, there’s a practice of using different quantization levels for K and V. I need to find the paper that analyzed which part (Key or Value) is more sensitive to quantization and causes more degradation…

If I’m mistaken, I’m sure more experienced members will correct me.

(Translated by Gemini)

14

Yeah, I wonder this too. I want to measure with and without and see the difference in scores (and time).

Unfortunately, I’ve not been having a good time trying to run these benchmarks. So many of them are very fragile and the results vary wildly. I even found that the TypeScript IfEvalCode ones are always 0% correctness for all models, because there is an error in some typescript that is stitched around what the model provides.

So for now, I’ve gone back to just a single benchmark (AgentBench) and have been running each sample 3 times and taking the mean score. I’ll update the repo later in the week with the scores (and as I find benchmarks that reproduce well, I’ll add them).

I’d really like to compare quantization on the kv cache, and things like autoround vs nfvp4. Currently everyone is focusing on the speed differences in them, but I can’t help but wonder whether those gains are really free. Right now, speed is not my biggest issue with the open models that fit on Spark, it’s getting them to do anything correctly 😄

15

There is likely no simple answer as to when FP8 cache degrades quality. Several factors need to be considered.

If we look back at weight quantization, we can’t simply say that a Q4 model is always worse than the base model. The real questions are: what specific model are we quantizing, what is the target task, and which quantization algorithm are we using?

There is an excellent article by RedHat: https://developers.redhat.com/articles/2026/02/04/accelerating-large-language-models-nvfp4-quantization#quantized_models_released .
I’d like to point out Figures 5 and 6. In Figure 6, all models showed performance degradation, meaning Q4 reduces accuracy for that specific task. However, Figure 5 is more interesting: some models maintained their accuracy, while others dropped significantly.

image
image276×750 27.8 KB

Here is my interpretation: Models within the same family (like Qwen3) are often trained on the same dataset. If you pour 15T tokens into both a 235B and a 30B model, the smaller model’s weights become ‘overtrained’ or ‘oversaturated.’ The knowledge is already heavily compressed. Thus, further quantization leads to a noticeable loss in quality.

NVIDIA and Unsloth love showing DeepSeek quantization charts, claiming ‘Q4 is safe.’ This is true for large, non-overtrained models. But look at the difference between Llama 3.1 and Llama 3.3 on the chart. Since Llama 3.3 had knowledge distilled from the 400B version, its weights are more ‘dense’ (overtrained), and we see degradation upon quantization again.

image
image269×550 24.7 KB

My conclusion would be: for small models (around 3B active parameters), it’s better to stick with FP8. For massive models, we often have no choice but to use Q4 just to fit them into a Spark cluster.

As for the choice between 4-bit formats: NVFP4 used to be slower than FP8, making it less practical for speed. Intel’s AutoRound (INT4) might have slightly lower quality but offers great throughput. Until NVFP4 becomes speed-competitive, I’d prefer Intel’s INT4.

Again, these are just my personal conclusions, and they might not be entirely accurate. I hope more experienced members of the community will correct me if I’m wrong.

(Translated by Gemini)

16

Ultimately, if generation speed isn’t a priority, it’s safer to avoid KV cache quantization altogether. My point isn’t that you should always quantize the cache; rather, it’s something we are occasionally forced to do.

We take that risk when we absolutely need the boost in throughput or need to save VRAM, but we must be aware of the potential trade-off in model quality.

However, this trade-off can vary depending on the situation — much like with weight quantization.

There is no simple answer like ‘FP8 cache is always safe.’ It’s the same as with Q4 for weights: sometimes it’s perfectly acceptable, and other times it significantly degrades quality.

Pinpointing and clearly defining exactly when this happens for the KV cache doesn’t seem to be a simple task.

(Translated by Gemini)

17

Yeah, to be clear - I wasn’t trying to answer the general question of whether to quantize kv cache (or whether nvfp4 is better than autoround). What I’d like to see is for a specific model, how do the benchmark scores vary for these things.

For example, if I review all the scores overall and decide the best model for my use case is Nemotron Super, it would be useful to see benchmark scores for that model in different kinds of quants, and with/without the kv cache quantized to see what I’m giving up for speed (or, how much better quality I can have if I give up a little speed).

I’m trying to replace the anecdotal “evidence” that X beats Y or that quantizing the kv cache has negligable effect with some actual numbers. If I can run the exact same model with/without kv cache quantization and see a couple of benchmark scores, that might help understand the quality trade being made for the performance difference.

18

However, for Nemotron-3, quantizing the KV cache doesn’t seem to make much sense. We don’t get any significant gains in either speed or VRAM capacity. Why run tests on it when we can maintain full precision with the full cache?

On the other hand, such a test would be very meaningful for MiniMax. In that case, if you avoid KV cache quantization, you lose a massive amount of context length and nearly half the speed. For a model like that, comparing quality is essential, because the risk of accuracy degradation is justified by the substantial benefits in performance and memory efficiency.

(Translated by Gemini)

19

Perhaps - it was just one I remembered had the kv cache flag noted on the instructions on the HF page:

--kv-cache-dtype fp8 \

There are many flags noted on HF pages, and I often wonder what impact they’re having on speed and/or quality.

20

I ran the same subset of AgentBench (3 times per sample, taking the mean result) across a bunch of models, including the dense Qwen3.6, and the results are a little surprising..

The bf16 dense model didn’t take much longer than the MoEs. I can only assume it got stuck less or had less failed tool calls or something (I haven’t dug into logs yet).

This is without dflash/prismquant or any of the other things to speed it up (it did have MTP=2, but so did the others). Whether this translates to real work, I don’t know.

Screenshot 2026-04-23 at 08.05.00
Screenshot 2026-04-23 at 08.05.00808×1050 62.8 KB