# Introducing Spark Auto Round /w OpenCode Instruct dataset

**URL:** <https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475>\
**Category:** DGX Spark / GB10\
**Tags:** spark, cuda, agentic-ai\
**Created:** [June 16, 2026, 8:48am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475 "2026-06-16T08:48:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 16, 2026, 8:48am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/1 "2026-06-16T08:48:05Z")

</div>

**An Int4 AutoRound quantization tool focused on the DGX Spark GB10 architecture**

[github.com/whpthomas/spark-auto-round](https://github.com/whpthomas/spark-auto-round)

# TLDR

Quantization seems to smooth out _‘knowledge’_ and _‘emergent behaviour’_ noise from the model. Using a dataset that closely matches your use case effectively _‘trains’_ the model by filtering out unwanted noise, improving test scores.

## Background

I have to choose a model to setup on client GB10 systems. This production model needs to be multi-modal. I also need some memory left over to run my main application.

Like many here I seem to be downloading new quantized models to my DGX Spark every week. The cycle is always the same: I swear this is my new daily driver, then there is an annoncement, a throng of forum activity and I grab the model from HuggingFace, serve it with vllm, and cross my fingers – this model be the one to rule them all!

There are some great models now, they work well for me, but are they the best choice for my clients? They code well or they don’t, but I never really understood why the good ones were better or _how well_ they really worked.

To be honest I really had no idea which quantization was optimal for my clients particular use case.

So I started digging.

## The Adaptive Sensitivity-Aware Quantization hypothesis

What if quantising sensitive layers were dragging down quality? I had seen a concept like this being discussed so I decided to experiment with it. The first thing was to generate quantization reports that detect problematic layers.

I tried implemeting Router KL Divergence, Hessian Trace Approximation, Layer-wise Peak Signal-to-Noise Ratio, Kurtosis & Outlier Detection, Fisher Information Trace but most of these tests either blew up memory or analysis time. So I settled on the cheap and cheerful combination of cosine similarity (perplexity) below 0.99 and Peak Signal-to-Noise Ratio (PSNR) under 45 dB, and sticking with dense models for now.

My hypothesis was that replacing sensitive layers with FP16 would improve model quality. I built `spark-asqa-substitute` — a utility that takes a working quantized model, identifies the sensitive layers, then swaps them out with FP16 from teh origianl model. The resulting output loaded and ran fine in vllm. A mixed INT4 + FP16 layer model is the best of both worlds – right?

The benchmark told a different story.

**Int4 AutoRound: 92/100**  
**Int4 AutoRound + ASAQ (layers 54, 58): 87/100**

Five points _lower_. Not higher. Why?

My best guess — those layers _are_ sensitive. But the Signed Gradient Decent algorithm had already found optimal rounding for them. Breaking that optimization by substituting FP16 degraded the model’s tool-calling ability. Maybe the vectors live in a different number space or magnitude? Maybe the quantizer knows what it’s doing? I don’t know, I didn’t see any improvement and down in this rabbit hole the light from the opening above me was getting dim.

This marked a turning point. Observing quality measures wasn’t just nice to have — it was the entire point.

## Observability changes everything

When I could _see_ per-layer sensitivity during quantization, the optimization process transformed from guesswork into engineering.

```auto
🟢 model.layers.0 Cosine 0.9965 PSNR 51.1 dB PASS
🟢 model.layers.1 Cosine 0.9967 PSNR 52.2 dB PASS
...
🟠 model.layers.22 Cosine 0.9912 PSNR 44.7 dB WARN
🟠 model.layers.23 Cosine 0.9883 PSNR 46.5 dB WARN

```

The `quantization-report.txt` tells you which layers to track. It doesn’t just say “this model is quantized” — it says _how well_ each layer survived the process.

So in parallel to ASAQ I also started iterating calibration datasets using the Qwen 3.5 0.8B model – which only takes 1:30 hours for each run, so at least I could get through a few iterations each day. My second hypothesis was that improvments on this smaller model would translate across to the larger 27B target.

First I tweaked all the auto-round settings, then I explored different datasets. The default dataset was `pile-10k` (Intel’s recommendation). So I tried `github-code-clean`, which was better, but optimal? So I tried `ultrachat_200k`. Each time, I’d quantize, benchmark, compare and track the layer error rates.

Then I added support for NVIDIA’s OpenCode Instruct dataset.

## This dataset changed the score

OpenCode Instruct is a 5M-sample dataset of coding instructions and solutions. The schema is rich: `input`, `output`, `unit_tests`, `llm_judgement`, `average_test_score`. For calibration, I concatenated `input` and `output`. This is a diverse code context that I was hoping would ‘teach’ the quantizer what matters for long context agentic toolcall workflows, not chat.

The results were significant.

**Qwen 3.5 0.8B — Tool Eval Bench scores by dataset:**

| Dataset | Score | Pass/Fail |
| --- | --- | --- |
| OpenCode Instruct | 69 | 41/13/15 |
| github-code-clean | 67 | 39/14/16 |
| pile-10k | 62 | 37/11/21 |

OpenCode Instruct outperformed all other models including the baseline fp16 test core of 67. But would this generalize?

**Qwen 3.6 27B with OpenCode Instruct: 92/100.**

The model passed 59 scenarios, got partial credit on 9, and failed 1 (a sleeper injection test that caught a real security issue) using MTP=3 speculative decoding (dflash=6 scores worse).

- Quantization command: `spark-auto-round --dataset "opencode-instruct" Qwen/Qwen3.6-27B`
- MTP averages ~26.4 t/s with `num_speculative_tokens: 3` for longer context and agentic coding
- DFlash averages ~38.1 t/s with `num_speculative_tokens: 6` for shorter context and instruction following

| # | Model | Scheme | Dataset | Score | t/s | Rating | P/F | Tokens |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 🥇 | **qwen3.6-27b-sar-oc-mpt** | **Int4** | OpenCode Instruct | **92** | 26.4 | ★★★★★ | 59/9/1 | 284K |
| 🥈 | qwen3.6-27b-sar-oc-dflash | Int4 | OpenCode Instruct | 90 | **38.1** | ★★★★★ | 57/10/2 | 265K |
| 🥉 | qwen/qwen3.6-27b-fp8 | fp8 | - | 88 | 18.1 | ★★★★ | 57/8/4 | 275K |
| 4 | qwen3.6-27b-sar-oc | Int4 | OpenCode Instruct | 88 | 12.5 | ★★★★ | 57/8/4 | 275K |
| 5 | qwen3.6-27b-sar-git-mtp | Int4 | Github Code Clean | 86 | 26.2 | ★★★★ | 54/10/5 | 268K |
| 6 | qwen/qwen3.6-27b | bf16 | - | 83 | 11.4 | ★★★★ | 53/9/7 | 243K |

I am optimistic.

## What spark-auto-round actually does

It’s a trimmed-down fork of Intel’s auto-round. I removed everything that isn’t needed by auto-round on CUDA with torch.compile:

- No GGUF, no MLX, no Gaudi HPU support
- No diffusion model evaluation
- No CPU fallback
- No 8-bit or 2-bit quantization

What’s left is a simple command line:

```bash
spark-auto-round Qwen/Qwen3.6-27B

```

One command. W4A16 quantization, optimal GB10 settings with torch.compile always enabled. The output goes to `./models/Qwen3.6-27B-int4-AutoRound/`.

A `quantization-report.txt` appears alongside the model weights.

The defaults are tuned for GB10 are:

| Setting | Default | Why |
| --- | --- | --- |
| `--iters` | 1000 | Enough for convergence, not so many it wastes time |
| `--nsamples` | 512 | Good coverage without memory pressure |
| `--seqlen` | 2048 | Balances context depth with calibration speed |
| `--batch_size` | 8 | Fits comfortably in 128GB unified memory |
| `--group_size` | 128 | Standard for INT4 quantization |
| `--dataset` | github-code-clean | Solid baseline, but use `opencode-instruct` for best results |

For large models that exceed memory:

```bash
spark-auto-round Qwen/Qwen3.5-122B-A10B --memory_utilization 75

```

The tool auto-detects whether the model fits in memory. If it doesn’t, it offloads blocks to disk and quantizes them one at a time. You control the threshold with `--memory_utilization` (50-95%).

## Why not NVFP4?

While NVFP4 suppport has improved significantly, I wanted a reference INT4 implementation for comparison. AutoRound is the most mature INT4 quantization method with vllm. By optimizing it specifically for GB10, I get a more reliable baseline for a true apples to apples comparison.

## What’s next

The quantization reports revealed that the last few layers in every model are consistently sensitive. I suspect this is a general property of transformer architectures — the output layers are inherently more sensitive to weight perturbation because they’re closer to the loss surface.

I’d like to explore whether this insight can inform better default settings. For proecessing really /newlarge models, maybe fewer iterations on early layers, more on late layers? Maybe different group sizes per layer? The observability makes these questions answerable.

## Try it

```bash
python -m venv .sar-venv
source .sar-venv/bin/activate
pip install git+https://github.com/whpthomas/spark-auto-round.git
spark-auto-round Qwen/Qwen3.6-27B --dataset opencode-instruct

```

The quantized model serves with vllm. The README has a complete docker command.

If you’re interested in experimenting with quantized models on your Spark, I’d love to hear your results. Open an issue, share your `quantization-report.txt`, tell me what’s working and what isn’t. The whole point of this tool is to make quantization results more observable. So we can understand and reason about them.

---

<div class="post-metadata">

**Author:** ![DColt](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@DColt](https://forums.developer.nvidia.com/u/DColt)\
**Post date:** [June 16, 2026, 3:43pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/2 "2026-06-16T15:43:10Z")

</div>

This looks interesting to try.

Have you attempted this on the Qwen3.6-35B-A3B model?  
I keep running into a crash whenever I run it.

> (.venv) dcolt@gx10-c046:~/spark-auto-round$ spark-auto-round Qwen/Qwen3.6-35B-A3B  
> 2026-06-16 17:40:35 INFO **main**.py L143: Spark Auto Round version 0.14.1  
> –batch\_size 8  
> –nsamples 512  
> –seqlen 2048  
> –group\_size 128  
> –iters 1000  
> –dataset github-code-clean  
> –output\_dir ./models  
> 2026-06-16 17:40:35 INFO **main**.py L169: `torch.compile` is enabled to reduce tuning costs. Disable with --disable\_torch\_compile if it causes issues.  
> 2026-06-16 17:40:35 INFO **main**.py L177: start to quantize Qwen/Qwen3.6-35B-A3B  
> 2026-06-16 17:40:35 WARNING memory.py L245: Cannot estimate parameter count from config — hidden\_size or intermediate\_size missing. Defaulting to 0 (will use whole-model path).  
> 2026-06-16 17:40:36 INFO memory.py L329: Memory analysis:  
> Model: Qwen/Qwen3.6-35B-A3B  
> Parameters: 0.0B  
> Model size: 0.0 GB (bfloat16)  
> Available: 66.6 GB  
> Threshold: 50.0 GB (75%)  
> Strategy: whole-model (fits in memory)  
> Blocks: 40 layers  
> Block size: ~0 MB each  
> Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF\_TOKEN to enable higher rate limits and faster downloads.  
> Fetching 26 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00\<00:00, 4809.98it/s]  
> Download complete: : 0.00B [00:00, ?B/s] [transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow [GitHub - fla-org/flash-linear-attention: 🚀 Efficient implementations for emerging model architectures · GitHub](https://github.com/fla-org/flash-linear-attention#installation) and [GitHub - Dao-AILab/causal-conv1d: Causal depthwise conv1d in CUDA, with a PyTorch interface · GitHub](https://github.com/Dao-AILab/causal-conv1d) | 0/26 [00:00\<?, ?it/s]  
> Download complete: : 0.00B [00:00, ?B/s]  
> Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 693/693 [00:00\<00:00, 9326.69it/s]  
> 2026-06-16 17:40:39 WARNING logging.py L340: some layers are skipped quantization (shape not divisible by 32): model.layers.[0-39].mlp.shared\_expert\_gate  
> 2026-06-16 17:40:39 INFO replace\_modules.py L120: Experts (before replacement) [model.layers.0.mlp.experts] (Qwen3\_5MoeExperts):  
> Qwen3\_5MoeExperts(  
> (act\_fn): SiLUActivation()  
> )  
> 2026-06-16 17:40:39 INFO memory.py L487: [Memory Monitor] Before applying custom replacements: ‘peak\_ram’: 1.02GB  
> 2026-06-16 17:40:39 INFO replace\_modules.py L360: Found 40 modules to replace  
> Replacing modules: 0%| | 0/40 [00:00\<?, ?it/s]  
> Traceback (most recent call last):  
> File “/home/dcolt/spark-auto-round/.venv/bin/spark-auto-round”, line 8, in  
> sys.exit(run())  
> ^^^^^  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/ **main**.py”, line 268, in run  
> start()  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/ **main**.py”, line 133, in start  
> tune(args)  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/ **main**.py”, line 261, in tune  
> model, folders = autoround.quantize\_and\_save(args.output\_dir, format=format)  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/compressors/base.py”, line 1248, in quantize\_and\_save  
> self.post\_init()  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/compressors/data\_driven.py”, line 122, in post\_init  
> super().post\_init()  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/compressors/base.py”, line 538, in post\_init  
> self.\_patch\_model()  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/compressors/base.py”, line 746, in \_patch\_model  
> self.model\_context.apply\_patches(self.formats)  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/context/model.py”, line 239, in apply\_patches  
> self.model = update\_module(  
> ^^^^^^^^^^^^^^  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/special\_model\_handler.py”, line 394, in update\_module  
> model = apply\_replacements(model)  
> ^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/modeling/fused\_moe/replace\_modules.py”, line 318, in apply\_replacements  
> \_apply\_custom\_replacements(model)  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/utils/device/memory.py”, line 488, in wrapper  
> result = func(\*args, \*\*kwargs)  
> ^^^^^^^^^^^^^^^^^^^^^  
> File “/home/dcolt/spark-auto-round/.venv/lib/python3.12/site-packages/auto\_round/modeling/fused\_moe/replace\_modules.py”, line 370, in \_apply\_custom\_replacements  
> with dump\_memory\_usage\_ctx(f"Replacing module {name}", log\_level=“debug”):  
> TypeError: ‘generator’ object does not support the context manager protocol

Running it on the Qwen3.6-27B seems to be working just fine.

---

<div class="post-metadata">

**Author:** ![miken](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/miken/32/513645_2.png) [@miken](https://forums.developer.nvidia.com/u/miken)\
**Post date:** [June 16, 2026, 3:53pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/3 "2026-06-16T15:53:17Z")

</div>

@whpthomas Have you thought about uploading your optimised quants to HF?

Could build up a full collection with a leaderboard helping users to pick the right fit for them.

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 16, 2026, 7:25pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/4 "2026-06-16T19:25:57Z")

</div>

> [@miken](#):
>
> Have you thought about uploading your optimised quants to HF?

I have but I live in a rural area in Australia, and the models are just to large to upload sucessfully.

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 16, 2026, 7:48pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/5 "2026-06-16T19:48:58Z")

</div>

> [@DColt](#):
>
> Qwen/Qwen3.6-35B-A3B

Thanks, I pushed a fix. I was planning to Quantize 35b next – there was an attribute pattern match missing.

I would recommend `spark-auto-round --dataset "opencode-instruct" Qwen/Qwen3.6-35B-A3B`

```auto
2026-06-17 05:36:36 INFO __main__.py L143: Spark Auto Round version 0.14.1
  --batch_size 8
  --nsamples 512
  --seqlen 2048
  --group_size 128
  --iters 1000
  --dataset opencode-instruct
  --output_dir ./models
2026-06-17 05:36:36 INFO __main__.py L169: `torch.compile` is enabled to reduce tuning costs. Disable with --disable_torch_compile if it causes issues.
2026-06-17 05:36:36 INFO __main__.py L177: start to quantize Qwen/Qwen3.6-35B-A3B
2026-06-17 05:36:37 INFO memory.py L336: Memory analysis:
  Model: Qwen/Qwen3.6-35B-A3B
  Parameters: 33.4B
  Model size: 62.2 GB (bfloat16)
  Available: 44.3 GB
  Threshold: 33.3 GB (75%)
  Strategy: block-offload (model exceeds threshold)
  Blocks: 40 layers
  Block size: ~1592 MB each
Fetching 26 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 3202.79it/s]
Download complete: : 0.00B [00:00, ?B/s] [transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Download complete: : 0.00B [00:00, ?B/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 693/693 [00:00<00:00, 9844.88it/s]
2026-06-17 05:36:41 WARNING logging.py L340: Skipped quantization for 40 layer(s) with shape not divisible by 32: model.layers.[0-39].mlp.shared_expert_gate. These layers will run in full precision (bf16) — this is expected for small/gate layers and has negligible impact.
2026-06-17 05:36:41 INFO replace_modules.py L120: Experts (before replacement) [model.layers.0.mlp.experts] (Qwen3_5MoeExperts):
Qwen3_5MoeExperts(
  (act_fn): SiLUActivation()
)
2026-06-17 05:36:41 INFO memory.py L495: [Memory Monitor] Before applying custom replacements: 'peak_ram': 0.99GB
2026-06-17 05:36:41 INFO replace_modules.py L360: Found 40 modules to replace
Replacing modules: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:01<00:00, 26.21it/s]
2026-06-17 05:36:43 INFO replace_modules.py L387: Replaced 40 modules
2026-06-17 05:36:43 INFO memory.py L498: [Memory Monitor] After applying custom replacements: 'peak_ram': 1.11GB
[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
2026-06-17 05:36:45 INFO replace_modules.py L120: Experts (after replacement) [model.layers.0.mlp.experts] (SequentialQwen3_5MoeExperts):
SequentialQwen3_5MoeExperts(
  (0-255): 256 x Qwen3_5MoeMLP(
    (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
    (up_proj): Linear(in_features=2048, out_features=512, bias=False)
    (down_proj): Linear(in_features=512, out_features=2048, bias=False)
    (act_fn): SiLUActivation()
  )
)
2026-06-17 05:36:45 INFO base.py L407: Using predefined ignore_layers: model.layers.[0-39].mlp.gate
2026-06-17 05:36:45 INFO utils.py L640: Ignored layers: model.layers.[0-39].mlp.gate
2026-06-17 05:36:46 INFO base.py L843: OffloadManager: offload mode (save to temp directory)
2026-06-17 05:36:49 INFO data_driven.py L693: start to cache block inputs
2026-06-17 05:36:49 INFO calib_dataset.py L1159: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00<00:00, 41555.78 examples/s]
Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00<00:00, 4413.81 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 544/544 [00:00<00:00, 969.31 examples/s]
Casting the dataset: 100%|██████████████████████████████████████████████████████████████████████████████████| 538/538 [00:01<00:00, 313.94 examples/s]
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00<00:00, 40618.49 examples/s]
Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00<00:00, 4305.61 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 544/544 [00:00<00:00, 973.91 examples/s]
Casting the dataset: 100%|██████████████████████████████████████████████████████████████████████████████████| 538/538 [00:01<00:00, 351.10 examples/s]
2026-06-17 05:37:11 INFO data_driven.py L716: caching done
Quantizing model.layers.0: 0%|░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|0/40 

```

---

<div class="post-metadata">

**Author:** ![redacted.design](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/redacted.design/32/504687_2.png) [@redacted.design](https://forums.developer.nvidia.com/u/redacted.design)\
**Post date:** [June 16, 2026, 8:38pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/6 "2026-06-16T20:38:48Z")

</div>

Fascinating work, aligns with a small project I have and now I’ll be doing some of it on the Spark because of it. Thank you very much for sharing.

---

<div class="post-metadata">

**Author:** ![bernardlbmi3](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bernardlbmi3/32/91999_2.png) [@bernardlbmi3](https://forums.developer.nvidia.com/u/bernardlbmi3)\
**Post date:** [June 16, 2026, 11:56pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/7 "2026-06-16T23:56:46Z")

</div>

Amazing to see how it is even better than the original FP16! How is that even possible! Mind blown. I am trying it on the Qwen3.6-35B-A3B model… Pointless since you are already doing it… but just for science.

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 17, 2026, 1:46am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/8 "2026-06-17T01:46:21Z")

</div>

> [@bernardlbmi3](#):
>
> Amazing to see how it is even better than the original FP16!

I really wasn’t expecting this. I don’t understand it either.

Possibilities might include:

- vllm is broken for fp16 and fp8
- tool eval bench is skewing results somehow
- fp16 has more instruction noise that quantization smooths out
- more kernal optimisation has been invested in serving quantized models
- The intel ‘inc’ kernel is just better written

I don’t know why, its part of the reason I released this early in development. These results don’t make sense. I was hoping for fb16 parity, I was not expecting to exceed it. This has happened with two Qwen models 0.8b and 27b – I am going to try 35b next. Eventually I want to try 3.5 122b

* * *

It will take about 14:30 hours to quantize 35b

 ![Screenshot from 2026-06-17 14-28-51](https://global.discourse-cdn.com/nvidia/original/4X/4/6/0/460e43796aada894b64a6023e2f1dfeb520fd83e.png)

---

<div class="post-metadata">

**Author:** ![DColt](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@DColt](https://forums.developer.nvidia.com/u/DColt)\
**Post date:** [June 17, 2026, 4:29am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/9 "2026-06-17T04:29:17Z")

</div>

> [@whpthomas](#):
>
> Thanks, I pushed a fix. I was planning to Quantize 35b next – there was an attribute pattern match missing.
> 
> I would recommend `spark-auto-round --dataset "opencode-instruct" Qwen/Qwen3.6-35B-A3B`

Awesome, it does seem to be running now. Thanks for the quick fix.

I’ll try to actually quantize it in the weekend.

---

<div class="post-metadata">

**Author:** ![paulsc.liu](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@paulsc.liu](https://forums.developer.nvidia.com/u/paulsc.liu)\
**Post date:** [June 17, 2026, 4:34am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/10 "2026-06-17T04:34:14Z")

</div>

Exellent work! You guys are going to push DGX Spark price higher (I mean it as a compliment)

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 17, 2026, 12:04pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/11 "2026-06-17T12:04:34Z")

</div>

I just finished Quantizing Qwen 3.6 35B for the first time. Here is the report. I am running benchmarks now.

```txt
=== Quantization Report ===
Model: Qwen/Qwen3.6-35B-A3B
Date: 2026-06-17 20:50:26
Version: 0.14.1

CLI Arguments:
  --batch_size 8
  --iters 1000
  --nsamples 512
  --seqlen 2048
  --dataset opencode-instruct
  --output_dir ./models/Qwen3.6-35B-A3B-int4-AutoRound

Memory Summary:
  Peak RAM: 41.69 GB
  Peak VRAM: 38.82 GB

Sensitivity Analysis:
─────────────────────────────────────────────────────────────────────────────────────────────────────
Layer Cosine Sim PSNR (dB) Iters uLoss Status
─────────────────────────────────────────────────────────────────────────────────────────────────────
🟢 model.layers.0 0.9994 62.3 684 11.14 → 0.48 PASS
🟢 model.layers.1 0.9994 63.1 947 3.36 → 0.74 PASS
🟢 model.layers.2 0.9994 67.0 993 6.66 → 1.26 PASS
🟢 model.layers.3 0.9990 62.1 421 8.40 → 2.31 PASS
🟢 model.layers.4 0.9989 61.7 760 5.99 → 2.75 PASS
🟢 model.layers.5 0.9987 61.0 883 8.06 → 3.54 PASS
🟢 model.layers.6 0.9988 72.1 950 8.76 → 4.07 PASS
🟢 model.layers.7 0.9979 67.8 944 14.31 → 6.46 PASS
🟢 model.layers.8 0.9977 66.4 765 13.33 → 7.38 PASS
🟢 model.layers.9 0.9977 65.6 970 15.10 → 8.80 PASS
🟢 model.layers.10 0.9981 80.4 898 23.48 → 10.74 PASS
🟢 model.layers.11 0.9975 79.2 995 29.37 → 13.89 PASS
🟢 model.layers.12 0.9973 79.1 946 23.84 → 14.30 PASS
🟢 model.layers.13 0.9969 78.1 812 26.28 → 17.91 PASS
🟢 model.layers.14 0.9973 77.6 798 33.64 → 20.28 PASS
🟢 model.layers.15 0.9955 75.9 674 50.15 → 28.37 PASS
🟢 model.layers.16 0.9953 75.2 967 55.53 → 33.30 PASS
🟢 model.layers.17 0.9946 74.6 467 59.47 → 35.88 PASS
🟢 model.layers.18 0.9945 73.3 839 79.88 → 51.00 PASS
🟢 model.layers.19 0.9924 71.7 362 104.94 → 71.95 PASS
🟢 model.layers.20 0.9926 71.7 555 98.18 → 73.29 PASS
🟢 model.layers.21 0.9923 71.4 824 117.57 → 74.99 PASS
🟢 model.layers.22 0.9915 69.7 907 186.60 → 106.11 PASS
🟠 model.layers.23 0.9894 69.1 783 192.03 → 131.24 WARN
🟠 model.layers.24 0.9892 69.1 885 166.91 → 130.72 WARN
🟠 model.layers.25 0.9885 68.5 310 196.08 → 150.31 WARN
🟢 model.layers.26 0.9900 68.2 584 217.66 → 158.01 PASS
🟠 model.layers.27 0.9880 67.4 382 269.98 → 186.95 WARN
🟠 model.layers.28 0.9893 67.3 786 286.51 → 204.30 WARN
🟠 model.layers.29 0.9893 67.3 828 299.38 → 198.03 WARN
🟠 model.layers.30 0.9900 66.4 739 327.99 → 235.18 WARN
🟠 model.layers.31 0.9882 49.0 882 424.14 → 302.32 WARN
🟠 model.layers.32 0.9879 47.8 783 456.14 → 326.36 WARN
🟠 model.layers.33 0.9893 48.7 231 631.92 → 421.33 WARN
🟠 model.layers.34 0.9892 58.2 806 826.51 → 561.06 WARN
🟠 model.layers.35 0.9889 49.0 663 1100.02 → 726.44 WARN
🟠 model.layers.36 0.9884 48.0 913 1188.17 → 872.80 WARN
🟠 model.layers.37 0.9897 48.3 482 1626.04 → 1067.69 WARN
🟢 model.layers.38 0.9905 55.5 503 2267.83 → 1566.12 PASS
🟢 model.layers.39 0.9906 54.4 335 3365.16 → 2042.25 PASS

Summary:
  Total blocks: 40
  Passed (🟢): 26
  Warning (🟠): 14

Thresholds: Cosine Similarity < 0.99, PSNR < 45.0 dB

```

---

<div class="post-metadata">

**Author:** ![azampatti](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/azampatti/32/503643_2.png) [@azampatti](https://forums.developer.nvidia.com/u/azampatti)\
**Post date:** [June 17, 2026, 3:26pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/12 "2026-06-17T15:26:06Z")

</div>

This is awesome, doc!

I will be quantizing 35B myself tonight and bench/compare it to my current usuals (122b hybrid and 35B FP8/NVFP4).

I’m curious on what difference do you see in your workload between the standard 122b-a10b-int4-AutoRound vs your custom quantized version of it or Albond’s Hybrid approach (if any)?

Thanks for this :)

---

<div class="post-metadata">

**Author:** ![bernardlbmi3](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bernardlbmi3/32/91999_2.png) [@bernardlbmi3](https://forums.developer.nvidia.com/u/bernardlbmi3)\
**Post date:** [June 17, 2026, 5:13pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/13 "2026-06-17T17:13:45Z")

</div>

Here is the spark-auto-round quantized version of Qwen3.6-35B-A3B:

[cyburn/Qwen3.6-35B-A3B-int4-AutoRound · Hugging Face](https://huggingface.co/cyburn/Qwen3.6-35B-A3B-int4-AutoRound)

DON’T use for now… not working… need some fixing.

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 17, 2026, 5:45pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/14 "2026-06-17T17:45:16Z")

</div>

Just in case you run into a problem loading – there was a regression error I fixed with the last change, but it depends which repo you pulled.

Here is a python script you can run: [fix-v14.1-layer-prefix.py](https://github.com/whpthomas/spark-auto-round/tree/master/scripts)

Once I have a chance to run the quantized model I will post the correct `json` files for 35b in a zip that you can drop in.

* * *

There are also scripts and example recipes in [spark-auto-round/spark-vllm-docker](https://github.com/whpthomas/spark-auto-round/tree/master/spark-vllm-docker) to launch local models using using community docker recipes with the `run-local-recipe.sh` script.

---

<div class="post-metadata">

**Author:** ![bernardlbmi3](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bernardlbmi3/32/91999_2.png) [@bernardlbmi3](https://forums.developer.nvidia.com/u/bernardlbmi3)\
**Post date:** [June 17, 2026, 6:14pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/15 "2026-06-17T18:14:33Z")

</div>

I am running in an issue… I think spark-auto-round is quantizing the ViT linear layers and this is breaking things in vllm:

Proper: exclude the ViT from quantization in AutoRound

The ViT linear layers (visual.\*) shouldn’t be quantized — they cause these kernel compatibility issues. In your AutoRound quantization call, add the ViT to the excluded layers. Let me check what the start script looks like.

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 17, 2026, 6:24pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/16 "2026-06-17T18:24:51Z")

</div>

I have a queue of jobs running at the moment, everything takes so long to run, I am working through the models systematically, I will let you know if I uncover anything else.

The 27b multi-modal model was working in my transcription pipeline. So I know it works, I just think I broke something fixing the 35b MOE.

---

<div class="post-metadata">

**Author:** ![mike136](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mike136/32/107586_2.png) [@mike136](https://forums.developer.nvidia.com/u/mike136)\
**Post date:** [June 17, 2026, 11:34pm UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/17 "2026-06-17T23:34:12Z")

</div>

I tried it on Qwen/Qwen3.5-122B-A10B, but it errored out..

(spark-auto-round) mike@pgx:~/spark-auto-round$ spark-auto-round --dataset “opencode-instruct” Qwen/Qwen3.5-122B-A10B  
2026-06-17 16:28:58 INFO **main**.py L143: Spark Auto Round version 0.14.1  
–batch\_size 8  
–nsamples 512  
–seqlen 2048  
–group\_size 128  
–iters 1000  
–dataset opencode-instruct  
–output\_dir ./models  
2026-06-17 16:28:58 INFO **main**.py L169: `torch.compile` is enabled to reduce tuning costs. Disable with --disable\_torch\_compile if it causes issues.  
2026-06-17 16:28:58 INFO **main**.py L177: start to quantize Qwen/Qwen3.5-122B-A10B  
2026-06-17 16:28:58 INFO memory.py L336: Memory analysis:  
Model: Qwen/Qwen3.5-122B-A10B  
Parameters: 118.5B  
Model size: 220.8 GB (bfloat16)  
Available: 39.3 GB  
Threshold: 29.4 GB (75%)  
Strategy: block-offload (model exceeds threshold)  
Blocks: 48 layers  
Block size: ~4710 MB each  
Fetching 39 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:00\<00:00, 10636.44it/s]  
Download complete: : 0.00B [00:00, ?B/s] [transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow [GitHub - fla-org/flash-linear-attention: 🚀 Efficient implementations for emerging model architectures · GitHub](https://github.com/fla-org/flash-linear-attention#installation) and [GitHub - Dao-AILab/causal-conv1d: Causal depthwise conv1d in CUDA, with a PyTorch interface · GitHub](https://github.com/Dao-AILab/causal-conv1d)  
Download complete: : 0.00B [00:00, ?B/s]  
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 831/831 [00:00\<00:00, 9529.46it/s]  
2026-06-17 16:29:01 WARNING logging.py L340: Skipped quantization for 48 layer(s) with shape not divisible by 32: model.layers.[0-47].mlp.shared\_expert\_gate. These layers will run in full precision (bf16) — this is expected for small/gate layers and has negligible impact.  
2026-06-17 16:29:01 INFO replace\_modules.py L120: Experts (before replacement) [model.layers.0.mlp.experts] (Qwen3\_5MoeExperts):  
Qwen3\_5MoeExperts(  
(act\_fn): SiLUActivation()  
)  
2026-06-17 16:29:01 INFO memory.py L495: [Memory Monitor] Before applying custom replacements: ‘peak\_ram’: 1.67GB  
2026-06-17 16:29:01 INFO replace\_modules.py L360: Found 48 modules to replace  
Replacing modules: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48/48 [00:01\<00:00, 28.71it/s]  
2026-06-17 16:29:03 INFO replace\_modules.py L387: Replaced 48 modules  
2026-06-17 16:29:03 INFO memory.py L498: [Memory Monitor] After applying custom replacements: ‘peak\_ram’: 1.82GB  
[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.  
2026-06-17 16:29:05 INFO replace\_modules.py L120: Experts (after replacement) [model.layers.0.mlp.experts] (SequentialQwen3\_5MoeExperts):  
SequentialQwen3\_5MoeExperts(  
(0-255): 256 x Qwen3\_5MoeMLP(  
(gate\_proj): Linear(in\_features=3072, out\_features=1024, bias=False)  
(up\_proj): Linear(in\_features=3072, out\_features=1024, bias=False)  
(down\_proj): Linear(in\_features=1024, out\_features=3072, bias=False)  
(act\_fn): SiLUActivation()  
)  
)  
2026-06-17 16:29:05 INFO base.py L407: Using predefined ignore\_layers: model.layers.[0-47].mlp.gate  
2026-06-17 16:29:05 INFO utils.py L640: Ignored layers: model.layers.[0-47].mlp.gate  
2026-06-17 16:29:07 INFO base.py L843: OffloadManager: offload mode (save to temp directory)  
2026-06-17 16:29:11 INFO data\_driven.py L693: start to cache block inputs  
2026-06-17 16:29:11 INFO calib\_dataset.py L1159: Preprocessing calibration dataset in a subprocess to avoid memory leaks…  
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00\<00:00, 45473.99 examples/s]  
Map: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00\<00:00, 4459.86 examples/s]  
Filter: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 544/544 [00:00\<00:00, 961.00 examples/s]  
Casting the dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 538/538 [00:01\<00:00, 469.81 examples/s]  
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00\<00:00, 44558.41 examples/s]  
Map: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2100/2100 [00:00\<00:00, 4645.20 examples/s]  
Filter: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 544/544 [00:00\<00:00, 972.70 examples/s]  
Casting the dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 538/538 [00:01\<00:00, 421.94 examples/s]  
Traceback (most recent call last):  
File “/home/mike/spark-auto-round/.venv/bin/spark-auto-round”, line 10, in  
sys.exit(run())  
^^^^^  
File “/home/mike/spark-auto-round/auto\_round/ **main**.py”, line 268, in run  
start()  
File “/home/mike/spark-auto-round/auto\_round/ **main**.py”, line 133, in start  
tune(args)  
File “/home/mike/spark-auto-round/auto\_round/ **main**.py”, line 261, in tune  
model, folders = autoround.quantize\_and\_save(args.output\_dir, format=format)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/auto\_round/compressors/base.py”, line 1265, in quantize\_and\_save  
self.quantize()  
File “/home/mike/spark-auto-round/auto\_round/compressors/data\_driven.py”, line 694, in quantize  
all\_inputs = self.try\_cache\_inter\_data\_gpucpu(  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 124, in decorate\_context  
return func(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/auto\_round/compressors/data\_driven.py”, line 146, in try\_cache\_inter\_data\_gpucpu  
return self.calibration.collect(block\_names, nsamples, layer\_names=layer\_names, last\_cache\_name=last\_cache\_name)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 124, in decorate\_context  
return func(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/auto\_round/calibration/llm.py”, line 79, in collect  
all\_inputs = self.cache\_inter\_data(  
^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 124, in decorate\_context  
return func(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/auto\_round/calibration/llm.py”, line 239, in cache\_inter\_data  
c.calib(nsamples, calib\_bs)  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 124, in decorate\_context  
return func(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/auto\_round/compressors/data\_driven.py”, line 169, in calib  
return self.calibration.calib(nsamples, bs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 124, in decorate\_context  
return func(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/auto\_round/calibration/llm.py”, line 387, in calib  
raise error  
File “/home/mike/spark-auto-round/auto\_round/calibration/llm.py”, line 366, in calib  
c.model(\*\*data\_new, \*\*kwargs)  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1778, in \_wrapped\_call\_impl  
return self.\_call\_impl(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1789, in \_call\_impl  
return forward\_call(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/transformers/utils/generic.py”, line 907, in wrapper  
output = func(self, \*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/transformers/models/qwen3\_5\_moe/modeling\_qwen3\_5\_moe.py”, line 1858, in forward  
output\_router\_logits if output\_router\_logits is not None else self.config.output\_router\_logits  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/mike/spark-auto-round/.venv/lib/python3.12/site-packages/transformers/configuration\_utils.py”, line 436, in **getattribute**  
return super(). **getattribute** (key)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
AttributeError: ‘Qwen3\_5MoeConfig’ object has no attribute ‘output\_router\_logits’  
(spark-auto-round) mike@pgx:~/spark-auto-round$

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 18, 2026, 4:30am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/18 "2026-06-18T04:30:03Z")

</div>

I am not a python dev – its like learning a foreign language to me – C and rust are my thing, so I am OK at coding but I really need agent help debugging. I think I got stuck in a loop trying to fix what I perceived was a brittle regex parser, but the error was actually in an overloaded output structure with _‘magic’_ hidden state. I had to implement a full --dry-run flag that regenerates the model config files just so I could trace the error through the codebase. It seems to be producing the correct json files now. I am running one last quantize run with the 0.8b model. When I confirm its working I have a 35b here that is also broken. So the dry-run should be able to rebuild its correct configs as well. When both those load and benchmark I will push a commit.

The good news is the --dry-run setting should fix broken config files in already quantized models. I believe shards were ok, the layer names were mangled.

> [@mike136](#):
>
> I tried it on Qwen/Qwen3.5-122B-A10B

Once the broken config files are fixed I will look into loading the larger 122b model.

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 19, 2026, 4:31am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/19 "2026-06-19T04:31:14Z")

</div>

**Some updates:**

I rebuilt the multi-modal `mllm` support in Spark Auto Round. Previously it just ignored the layers and left them at fb16, but for whatever reason this was working and it broke. So I just copied the upstream auto-round handler code over, wired it in and that appears to have fixed the vision layers in Qwen. I am currently quantizing 35b, it has about 4 hours to go. When its done I will benchmark, test visions layers and report back.

 ![Screenshot from 2026-06-19 14-02-47](https://global.discourse-cdn.com/nvidia/original/4X/9/c/9/9c9e8e9e56258b5ac0e53a9d1b03f084ebb5083f.png)

I still don’t trust the results I am seeing. I went back an tweaked all the vllm recipes. I managed to get better fp16 scores out of the Qwen 3.5 0.8b test model. Nevertheless the Spark Auto Round with `nvidia/OpenCode Instruct` dataset still repeatedly out performs fp16 on this model by 1%.

The real kicker is that SAR is 7% better than the intel auto-round.

 ![Screenshot 2026-06-19 at 2.02.30 pm](https://global.discourse-cdn.com/nvidia/original/4X/4/4/d/44d4e1cf5e3c777e6c829f477dde09c0baccb2f8.jpeg)

**Adaptive Sensitivity-Aware Quantization (ASAQ)**

I still haven’t given up on this idea. Maybe vllm was mis-configured. Maybe with the right recipe it will work. It is integrated into the repo. I am working on documentation for it now. It is a seperate cli tool. You run it the same way `spark-asqa-substitute <model>`. I reads the `quantization-report.csv` and substitutes sensitive layers with fp16 layers from the original model. For MOE it performs Router Jaccard Similarity to detect routing instability caused by quantization. The router (gating layer) decides which experts process each token. Quantization noise can flip these decisions, causing cascading quality degradation even if individual expert layers have high cosine similarity.

I just need 35b to test this with… Stay tuned `;)`

---

<div class="post-metadata">

**Author:** ![whpthomas](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/whpthomas/32/469595_2.png) [@whpthomas](https://forums.developer.nvidia.com/u/whpthomas)\
**Post date:** [June 19, 2026, 11:35am UTC](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475/20 "2026-06-19T11:35:20Z")

</div>

Here are some preliminary results. I only did one rep of each. Adding `--dtype=bfloat16` resulted in the highest score. Prefix caching with speculative decoding is still broken.

## Results with Qwen 3.6 35b a3b

| # | Model | Scheme | Dataset | Score | t/s | Rating | P/F | Tokens |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 🥇 | qwen3.6-35b-sar-bf16 | int4 | OpenCode Instruct | 93 | 65.4 | ★★★★★ | 60/8/1 | 275K |
| 🥈 | qwen3.6-35b-sar-mtp | int4 | OpenCode Instruct | 91 | 78.7 | ★★★★★ | 58/10/1 | 272K |
| 🥉 | qwen3.6-35b-sar | int4 | OpenCode Instruct | 91 | 64.7 | ★★★★★ | 59/8/2 | 283K |
| 4 | qwen3.6-35b-sar-pc | int4 | OpenCode Instruct | 91 | 64.5 | ★★★★★ | 59/8/2 | 284K |
| 5 | qwen/qwen3.6-35b | bf16 | - | 91 | 28.3 | ★★★★★ | 59/8/2 | 292K |
| 6 | qwen/qwen3.6-35b-fp8 | fp8 | - | 90 | 48.0 | ★★★★★ | 58/8/3 | 264K |
| 7 | qwen3.6-35b-sar-dflash | int4 | OpenCode Instruct | 88 | 97.1 | ★★★★ | 55/11/3 | 282K |
| 8 | qwen3.6-35b-sar-pc-mtp-bf16 | int4 | OpenCode Instruct | 47 | 73.4 | ★★ | 28/9/32 | 136K |
| 9 | qwen3.6-35b-sar-pc-mtp | int4 | OpenCode Instruct | 44 | 109.1 | ★★ | 26/9/34 | 131K |
| 10 | qwen3.6-35b-sar-pc-dflash | int4 | OpenCode Instruct | 43 | 102.3 | ★★ | 25/9/35 | 133K |

### Legend

- `-sar` Spark Auto Round
- `-pc` Prefix Caching
- `-mtp` MTP=2 Speculative Decoding (MTP=1 with `-pc`)
- `-dflash` DFlash=5 Speculative Decoding
- `-bf16` bfloat16 data type

[Next page](https://forums.developer.nvidia.com/t/introducing-spark-auto-round-w-opencode-instruct-dataset/373475.md?page=2)