An Int4 AutoRound quantization tool focused on the DGX Spark GB10 architecture
github.com/whpthomas/spark-auto-round
TLDR
Quantization seems to smooth out ‘knowledge’ and ‘emergent behaviour’ noise from the model. Using a dataset that closely matches your use case effectively ‘trains’ the model by filtering out unwanted noise, improving test scores.
Background
I have to choose a model to setup on client GB10 systems. This production model needs to be multi-modal. I also need some memory left over to run my main application.
Like many here I seem to be downloading new quantized models to my DGX Spark every week. The cycle is always the same: I swear this is my new daily driver, then there is an annoncement, a throng of forum activity and I grab the model from HuggingFace, serve it with vllm, and cross my fingers – this model be the one to rule them all!
There are some great models now, they work well for me, but are they the best choice for my clients? They code well or they don’t, but I never really understood why the good ones were better or how well they really worked.
To be honest I really had no idea which quantization was optimal for my clients particular use case.
So I started digging.
The Adaptive Sensitivity-Aware Quantization hypothesis
What if quantising sensitive layers were dragging down quality? I had seen a concept like this being discussed so I decided to experiment with it. The first thing was to generate quantization reports that detect problematic layers.
I tried implemeting Router KL Divergence, Hessian Trace Approximation, Layer-wise Peak Signal-to-Noise Ratio, Kurtosis & Outlier Detection, Fisher Information Trace but most of these tests either blew up memory or analysis time. So I settled on the cheap and cheerful combination of cosine similarity (perplexity) below 0.99 and Peak Signal-to-Noise Ratio (PSNR) under 45 dB, and sticking with dense models for now.
My hypothesis was that replacing sensitive layers with FP16 would improve model quality. I built spark-asqa-substitute — a utility that takes a working quantized model, identifies the sensitive layers, then swaps them out with FP16 from teh origianl model. The resulting output loaded and ran fine in vllm. A mixed INT4 + FP16 layer model is the best of both worlds – right?
The benchmark told a different story.
Int4 AutoRound: 92/100
Int4 AutoRound + ASAQ (layers 54, 58): 87/100
Five points lower. Not higher. Why?
My best guess — those layers are sensitive. But the Signed Gradient Decent algorithm had already found optimal rounding for them. Breaking that optimization by substituting FP16 degraded the model’s tool-calling ability. Maybe the vectors live in a different number space or magnitude? Maybe the quantizer knows what it’s doing? I don’t know, I didn’t see any improvement and down in this rabbit hole the light from the opening above me was getting dim.
This marked a turning point. Observing quality measures wasn’t just nice to have — it was the entire point.
Observability changes everything
When I could see per-layer sensitivity during quantization, the optimization process transformed from guesswork into engineering.
🟢 model.layers.0 Cosine 0.9965 PSNR 51.1 dB PASS
🟢 model.layers.1 Cosine 0.9967 PSNR 52.2 dB PASS
...
🟠 model.layers.22 Cosine 0.9912 PSNR 44.7 dB WARN
🟠 model.layers.23 Cosine 0.9883 PSNR 46.5 dB WARN
The quantization-report.txt tells you which layers to track. It doesn’t just say “this model is quantized” — it says how well each layer survived the process.
So in parallel to ASAQ I also started iterating calibration datasets using the Qwen 3.5 0.8B model – which only takes 1:30 hours for each run, so at least I could get through a few iterations each day. My second hypothesis was that improvments on this smaller model would translate across to the larger 27B target.
First I tweaked all the auto-round settings, then I explored different datasets. The default dataset was pile-10k (Intel’s recommendation). So I tried github-code-clean, which was better, but optimal? So I tried ultrachat_200k. Each time, I’d quantize, benchmark, compare and track the layer error rates.
Then I added support for NVIDIA’s OpenCode Instruct dataset.
This dataset changed the score
OpenCode Instruct is a 5M-sample dataset of coding instructions and solutions. The schema is rich: input, output, unit_tests, llm_judgement, average_test_score. For calibration, I concatenated input and output. This is a diverse code context that I was hoping would ‘teach’ the quantizer what matters for long context agentic toolcall workflows, not chat.
The results were significant.
Qwen 3.5 0.8B — Tool Eval Bench scores by dataset:
| Dataset | Score | Pass/Fail |
|---|---|---|
| OpenCode Instruct | 69 | 41/13/15 |
| github-code-clean | 67 | 39/14/16 |
| pile-10k | 62 | 37/11/21 |
OpenCode Instruct outperformed all other models including the baseline fp16 test core of 67. But would this generalize?
Qwen 3.6 27B with OpenCode Instruct: 92/100.
The model passed 59 scenarios, got partial credit on 9, and failed 1 (a sleeper injection test that caught a real security issue) using MTP=3 speculative decoding (dflash=6 scores worse).
- Quantization command:
spark-auto-round --dataset "opencode-instruct" Qwen/Qwen3.6-27B - MTP averages ~26.4 t/s with
num_speculative_tokens: 3for longer context and agentic coding - DFlash averages ~38.1 t/s with
num_speculative_tokens: 6for shorter context and instruction following
| # | Model | Scheme | Dataset | Score | t/s | Rating | P/F | Tokens |
|---|---|---|---|---|---|---|---|---|
| 🥇 | qwen3.6-27b-sar-oc-mpt | Int4 | OpenCode Instruct | 92 | 26.4 | ★★★★★ | 59/9/1 | 284K |
| 🥈 | qwen3.6-27b-sar-oc-dflash | Int4 | OpenCode Instruct | 90 | 38.1 | ★★★★★ | 57/10/2 | 265K |
| 🥉 | qwen/qwen3.6-27b-fp8 | fp8 | - | 88 | 18.1 | ★★★★ | 57/8/4 | 275K |
| 4 | qwen3.6-27b-sar-oc | Int4 | OpenCode Instruct | 88 | 12.5 | ★★★★ | 57/8/4 | 275K |
| 5 | qwen3.6-27b-sar-git-mtp | Int4 | Github Code Clean | 86 | 26.2 | ★★★★ | 54/10/5 | 268K |
| 6 | qwen/qwen3.6-27b | bf16 | - | 83 | 11.4 | ★★★★ | 53/9/7 | 243K |
I am optimistic.
What spark-auto-round actually does
It’s a trimmed-down fork of Intel’s auto-round. I removed everything that isn’t needed by auto-round on CUDA with torch.compile:
- No GGUF, no MLX, no Gaudi HPU support
- No diffusion model evaluation
- No CPU fallback
- No 8-bit or 2-bit quantization
What’s left is a simple command line:
spark-auto-round Qwen/Qwen3.6-27B
One command. W4A16 quantization, optimal GB10 settings with torch.compile always enabled. The output goes to ./models/Qwen3.6-27B-int4-AutoRound/.
A quantization-report.txt appears alongside the model weights.
The defaults are tuned for GB10 are:
| Setting | Default | Why |
|---|---|---|
--iters |
1000 | Enough for convergence, not so many it wastes time |
--nsamples |
512 | Good coverage without memory pressure |
--seqlen |
2048 | Balances context depth with calibration speed |
--batch_size |
8 | Fits comfortably in 128GB unified memory |
--group_size |
128 | Standard for INT4 quantization |
--dataset |
github-code-clean | Solid baseline, but use opencode-instruct for best results |
For large models that exceed memory:
spark-auto-round Qwen/Qwen3.5-122B-A10B --memory_utilization 75
The tool auto-detects whether the model fits in memory. If it doesn’t, it offloads blocks to disk and quantizes them one at a time. You control the threshold with --memory_utilization (50-95%).
Why not NVFP4?
While NVFP4 suppport has improved significantly, I wanted a reference INT4 implementation for comparison. AutoRound is the most mature INT4 quantization method with vllm. By optimizing it specifically for GB10, I get a more reliable baseline for a true apples to apples comparison.
What’s next
The quantization reports revealed that the last few layers in every model are consistently sensitive. I suspect this is a general property of transformer architectures — the output layers are inherently more sensitive to weight perturbation because they’re closer to the loss surface.
I’d like to explore whether this insight can inform better default settings. For proecessing really /newlarge models, maybe fewer iterations on early layers, more on late layers? Maybe different group sizes per layer? The observability makes these questions answerable.
Try it
python -m venv .sar-venv
source .sar-venv/bin/activate
pip install git+https://github.com/whpthomas/spark-auto-round.git
spark-auto-round Qwen/Qwen3.6-27B --dataset opencode-instruct
The quantized model serves with vllm. The README has a complete docker command.
If you’re interested in experimenting with quantized models on your Spark, I’d love to hear your results. Open an issue, share your quantization-report.txt, tell me what’s working and what isn’t. The whole point of this tool is to make quantization results more observable. So we can understand and reason about them.
