KAT Coder v2.5 Dev versus Ornith 1.0 35B - Spark AutoRound

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Excited to post about 2 topics. Quantization and Inference - [Ornith 1.0 model](azampatti/Ornith-1.0-35B-int4-AutoRound-SAR) using the [Spark-AutoRound](GitHub - whpthomas/spark-auto-round · GitHub) quantization by @whpthomas has been my favorite combination of Speed and Quality for Inference on my Asus GB10.

When I saw that KAT 2.5 Dev was flexing on Ornith in the benchmark graph I was intrigued. KAT was trained on the intention to have more efficient thinking and fewer tool calls to accomplish the same or better level of work as similar models in its parameter class. I have not seen this model mentioned in the forum so I thought this was a good opportunity to do my first quantization and share.

kat-benchmark
kat-benchmark763×371 65.8 KB

Here is KAT 2.5 Dev with the Qwen 3.6 MTP headers added in. Its language only so no vision. I used the opencode spark autoround method mentioned at the top which delivers int4 speed at fp8 quality.

https://huggingface.co/slopops/KAT-Coder-V2.5-Dev-MTP-int4-AutoRound-SAR

Testing: Using the same vllm settings and only switching between Ornith and KAT models, I was able to do a benchmark comparison on tool eval and also a coding/planning task through a harness.

Results: My opinion is that Ornith is a superior model for my coding use case. It got a 84 on tool eval compare to Ornith at 87. Also Ornith was much more thorough on planning out a task. KAT also encountered more issues with tool calling and had a harder time implementing the plan.

Please feel free to try out the quant - it is quick (was getting over 85 t/s accepted) and I believe solid quality. I am sure someone could suggest better settings or have an idea I missed!

2

Nice work!

3

Neat! I’m going to include this in my benchmark soon (I’m finishing a project on my GB10 that is taking weeks) to see how it ranks!

Spoiler alert, my project includes heavy use of Spark-AutoRound as well :)

4

I’m not sure how they got those scores - I just finishing running some benchmarks on it (bfcl, bigcodebench, ifevalcode) and it was pretty bad:

Although, I’ll note that I have very little faith in these benchmarks lately. I’ve dropped so many from my table because I started looking into failures and found that many of them were outright wrong (for example typos in the validation that require the model to get the answer wrong to be scored 🙃).

5

I am running another model and I ran into a quality issue with the checkpoint resume functionality in spark autoround. The run failed overnight at 21/48. I did a resume in the morning and you can see the quality of the resume is not great and getting worse (I eventually killed it). Here is the report to show the quality from my first checkpoint resume.

2026-08-11 13:47:08 INFO data_driven.py L803: Found valid checkpoint: 21/40 blocks completed in ./models/BigBang-v1-int4-AutoRound/.cache
2026-08-11 13:47:08 INFO data_driven.py L1376: Loading 21 completed blocks from checkpoint...
🟢 model.language_model.layers.1 | Cosine similarity 0.9994 | Peak Signal-to-Noise Ratio 62.5 | uloss 11 → 1 | iter 982/983
Quantizing model.layers.1:  2%|████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|1/40 [0:02<1:48, 2.8s/it]
🟢 model.language_model.layers.2 | Cosine similarity 0.9995 | Peak Signal-to-Noise Ratio 64.9 | uloss 3 → 1 | iter 946/947
Quantizing model.layers.2:  5%|████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|2/40 [0:04<1:16, 2.0s/it]
🟢 model.language_model.layers.3 | Cosine similarity 0.9994 | Peak Signal-to-Noise Ratio 66.4 | uloss 7 → 1 | iter 728/729
Quantizing model.layers.3:  7%|████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|3/40 [0:05<1:06, 1.8s/it]
🟢 model.language_model.layers.4 | Cosine similarity 0.9992 | Peak Signal-to-Noise Ratio 63.1 | uloss 8 → 2 | iter 757/758
Quantizing model.layers.4: 10%|████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|4/40 [0:06<0:59, 1.7s/it]
🟢 model.language_model.layers.5 | Cosine similarity 0.9991 | Peak Signal-to-Noise Ratio 62.5 | uloss 6 → 3 | iter 999/1000
Quantizing model.layers.5: 12%|█████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|5/40 [0:07<0:55, 1.6s/it]
🟢 model.language_model.layers.6 | Cosine similarity 0.9989 | Peak Signal-to-Noise Ratio 61.8 | uloss 8 → 4 | iter 949/950
Quantizing model.layers.6: 15%|█████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|6/40 [0:09<0:51, 1.5s/it]
🟢 model.language_model.layers.7 | Cosine similarity 0.9988 | Peak Signal-to-Noise Ratio 72.2 | uloss 9 → 5 | iter 873/874
Quantizing model.layers.7: 17%|█████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|7/40 [0:10<0:49, 1.5s/it]
🟢 model.language_model.layers.8 | Cosine similarity 0.9982 | Peak Signal-to-Noise Ratio 68.5 | uloss 16 → 7 | iter 913/914
Quantizing model.layers.8: 20%|█████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|8/40 [0:11<0:46, 1.5s/it]
🟢 model.language_model.layers.9 | Cosine similarity 0.9981 | Peak Signal-to-Noise Ratio 67.3 | uloss 14 → 8 | iter 989/990
Quantizing model.layers.9: 22%|█████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|9/40 [0:12<0:44, 1.4s/it]
🟢 model.language_model.layers.10 | Cosine similarity 0.9980 | Peak Signal-to-Noise Ratio 66.4 | uloss 15 → 9 | iter 974/975
Quantizing model.layers.10: 25%|█████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|10/40 [0:14<0:42, 1.4s/it]
🟢 model.language_model.layers.11 | Cosine similarity 0.9983 | Peak Signal-to-Noise Ratio 81.0 | uloss 25 → 12 | iter 841/842
Quantizing model.layers.11: 27%|█████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|11/40 [0:15<0:40, 1.4s/it]
🟢 model.language_model.layers.12 | Cosine similarity 0.9978 | Peak Signal-to-Noise Ratio 80.0 | uloss 32 → 15 | iter 994/995
Quantizing model.layers.12: 30%|█████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|12/40 [0:16<0:38, 1.4s/it]
🟢 model.language_model.layers.13 | Cosine similarity 0.9975 | Peak Signal-to-Noise Ratio 79.8 | uloss 26 → 15 | iter 945/946
Quantizing model.layers.13: 32%|█████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|13/40 [0:17<0:37, 1.4s/it]
🟢 model.language_model.layers.14 | Cosine similarity 0.9971 | Peak Signal-to-Noise Ratio 78.8 | uloss 29 → 19 | iter 977/978
Quantizing model.layers.14: 35%|██████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|14/40 [0:19<0:35, 1.4s/it]
🟢 model.language_model.layers.15 | Cosine similarity 0.9973 | Peak Signal-to-Noise Ratio 78.3 | uloss 36 → 22 | iter 975/976
Quantizing model.layers.15: 37%|██████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|15/40 [0:20<0:33, 1.3s/it]
🟢 model.language_model.layers.16 | Cosine similarity 0.9958 | Peak Signal-to-Noise Ratio 76.7 | uloss 50 → 30 | iter 673/674
Quantizing model.layers.16: 40%|██████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|16/40 [0:21<0:32, 1.3s/it]
🟢 model.language_model.layers.17 | Cosine similarity 0.9954 | Peak Signal-to-Noise Ratio 76.0 | uloss 56 → 35 | iter 885/886
Quantizing model.layers.17: 42%|██████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|17/40 [0:22<0:30, 1.3s/it]
🟢 model.language_model.layers.18 | Cosine similarity 0.9947 | Peak Signal-to-Noise Ratio 75.4 | uloss 61 → 39 | iter 466/467
Quantizing model.layers.18: 45%|██████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|18/40 [0:23<0:29, 1.3s/it]
🟢 model.language_model.layers.19 | Cosine similarity 0.9945 | Peak Signal-to-Noise Ratio 74.2 | uloss 77 → 52 | iter 868/869
Quantizing model.layers.19: 47%|██████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|19/40 [0:25<0:27, 1.3s/it]
🟢 model.language_model.layers.20 | Cosine similarity 0.9925 | Peak Signal-to-Noise Ratio 72.6 | uloss 114 → 77 | iter 840/841
Quantizing model.layers.20: 50%|███████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|20/40 [0:26<0:26, 1.3s/it]
2026-08-11 13:47:33 INFO data_driven.py L1415: Completed block loading. Resuming from block 21.
2026-08-11 13:47:33 INFO data_driven.py L1458: Skipping 21 of 40 completed layers in model.language_model.layers.0
W0811 13:47:40.342000 1 torch/_inductor/utils.py:1934] [4/0_1] Not enough SMs to use max_autotune_gemm mode
2026-08-11 14:04:06 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 1050 -> iter 946: 231
2026-08-11 14:04:19 INFO memory.py L481: 'peak_ram': 27.87GB, 'peak_vram': 25.62GB
🟢 model.language_model.layers.21 | Cosine similarity 0.9908 | Peak Signal-to-Noise Ratio 61.9 | uloss 1050 → 231 | iter 946/947
Quantizing model.layers.21: 52%|█████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|21/40 [17:15<15:36, 49.3s/it]
2026-08-11 14:19:52 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 3695 -> iter 974: 1988
2026-08-11 14:20:03 INFO memory.py L481: 'peak_ram': 30.48GB, 'peak_vram': 27.16GB
🟠 model.language_model.layers.22 | Cosine similarity 0.9770 | Peak Signal-to-Noise Ratio 52.2 | uloss 3695 → 1988 | iter 974/975
Quantizing model.layers.22: 55%|█████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|22/40 [32:58<26:59, 89.9s/it]
/usr/local/lib/python3.12/dist-packages/torch/autograd/graph.py:979: UserWarning: Flash Attention defaults to a non-deterministic algorithm. To explicitly enable determinism call torch.use_deterministic_algorithms(True, warn_only=False). (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/cuda/attention_backward.cu:125.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2026-08-11 14:34:35 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 5764 -> iter 765: 4053
2026-08-11 14:34:43 INFO memory.py L481: 'peak_ram': 32.87GB, 'peak_vram': 27.32GB
🟠 model.language_model.layers.23 | Cosine similarity 0.9610 | Peak Signal-to-Noise Ratio 49.9 | uloss 5764 → 4053 | iter 765/766
Quantizing model.layers.23: 57%|█████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|23/40 [47:39<35:13, 124.3s/it]
2026-08-11 14:50:11 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 6786 -> iter 61: 5306
2026-08-11 14:50:21 INFO memory.py L481: 'peak_ram': 35.21GB, 'peak_vram': 27.32GB
🟠 model.language_model.layers.24 | Cosine similarity 0.9519 | Peak Signal-to-Noise Ratio 50.0 | uloss 6786 → 5306 | iter 61/62
Quantizing model.layers.24: 60%|████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|24/40 [1:03:17<42:11, 158.2s/it]
2026-08-11 15:05:39 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 10591 -> iter 753: 7950
2026-08-11 15:05:49 INFO memory.py L481: 'peak_ram': 35.21GB, 'peak_vram': 27.32GB
🟠 model.language_model.layers.25 | Cosine similarity 0.9356 | Peak Signal-to-Noise Ratio 48.3 | uloss 10591 → 7950 | iter 753/754
Quantizing model.layers.25: 62%|████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|25/40 [1:18:44<47:14, 189.0s/it]
2026-08-11 15:21:11 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 14885 -> iter 929: 9966
2026-08-11 15:21:21 INFO memory.py L481: 'peak_ram': 35.36GB, 'peak_vram': 27.32GB
🟠 model.language_model.layers.26 | Cosine similarity 0.9116 | Peak Signal-to-Noise Ratio 46.9 | uloss 14885 → 9966 | iter 929/930
Quantizing model.layers.26: 65%|████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|26/40 [1:34:16<50:45, 217.6s/it]
2026-08-11 15:35:47 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 12763 -> iter 367: 10299
2026-08-11 15:35:55 INFO memory.py L481: 'peak_ram': 37.68GB, 'peak_vram': 27.43GB
🟠 model.language_model.layers.27 | Cosine similarity 0.9077 | Peak Signal-to-Noise Ratio 47.3 | uloss 12763 → 10299 | iter 367/368
Quantizing model.layers.27: 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|27/40 [1:48:51<52:24, 241.9s/it]
2026-08-11 15:51:18 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 13523 -> iter 657: 10988
2026-08-11 15:51:28 INFO memory.py L481: 'peak_ram': 37.68GB, 'peak_vram': 27.43GB
🟠 model.language_model.layers.28 | Cosine similarity 0.8974 | Peak Signal-to-Noise Ratio 46.9 | uloss 13523 → 10988 | iter 657/658
Quantizing model.layers.28: 70%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|28/40 [2:04:24<53:19, 266.6s/it]
2026-08-11 16:06:56 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 18507 -> iter 73: 10856
2026-08-11 16:07:07 INFO memory.py L481: 'peak_ram': 37.82GB, 'peak_vram': 27.43GB

I then did a wipe and fresh try again at it. Here is the quality on this new attempt. As you can see it is MUCH better:

2026-08-11 16:34:44 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 11 -> iter 939: 1
2026-08-11 16:34:57 INFO memory.py L481: 'peak_ram': 12.27GB, 'peak_vram': 25.96GB
🟢 model.language_model.layers.0 | Cosine similarity 0.9994 | Peak Signal-to-Noise Ratio 62.7 | uloss 11 → 1 | iter 939/940
Quantizing model.layers.1:  2%|████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|1/40 [17:04<11:05:49, 1024.3s/it]
2026-08-11 16:50:28 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 3 -> iter 946: 1
2026-08-11 16:50:38 INFO memory.py L481: 'peak_ram': 14.85GB, 'peak_vram': 27.98GB
🟢 model.language_model.layers.1 | Cosine similarity 0.9995 | Peak Signal-to-Noise Ratio 64.8 | uloss 3 → 1 | iter 946/947
Quantizing model.layers.2:  5%|████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|2/40 [32:45<10:22:28, 982.9s/it]
2026-08-11 17:05:55 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 7 -> iter 962: 1
2026-08-11 17:06:05 INFO memory.py L481: 'peak_ram': 17.23GB, 'peak_vram': 27.98GB
🟢 model.language_model.layers.2 | Cosine similarity 0.9995 | Peak Signal-to-Noise Ratio 66.6 | uloss 7 → 1 | iter 962/963
Quantizing model.layers.3:  7%|████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|3/40 [48:12<9:54:37, 964.3s/it]
/usr/local/lib/python3.12/dist-packages/torch/autograd/graph.py:979: UserWarning: Flash Attention defaults to a non-deterministic algorithm. To explicitly enable determinism call torch.use_deterministic_algorithms(True, warn_only=False). (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/cuda/attention_backward.cu:125.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2026-08-11 17:20:29 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 8 -> iter 757: 2
2026-08-11 17:20:38 INFO memory.py L481: 'peak_ram': 19.61GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.3 | Cosine similarity 0.9991 | Peak Signal-to-Noise Ratio 63.0 | uloss 8 → 2 | iter 757/758
Quantizing model.layers.4: 10%|████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|4/40 [1:02:45<9:24:49, 941.4s/it]
2026-08-11 17:36:03 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 6 -> iter 930: 3
2026-08-11 17:36:13 INFO memory.py L481: 'peak_ram': 19.61GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.4 | Cosine similarity 0.9991 | Peak Signal-to-Noise Ratio 62.5 | uloss 6 → 3 | iter 930/931
Quantizing model.layers.5: 12%|████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|5/40 [1:18:20<9:08:24, 940.1s/it]
2026-08-11 17:51:33 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 8 -> iter 949: 4
2026-08-11 17:51:43 INFO memory.py L481: 'peak_ram': 19.61GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.5 | Cosine similarity 0.9989 | Peak Signal-to-Noise Ratio 61.8 | uloss 8 → 4 | iter 949/950
Quantizing model.layers.6: 15%|████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|6/40 [1:33:50<8:51:44, 938.4s/it]
2026-08-11 18:07:01 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 9 -> iter 873: 5
2026-08-11 18:07:12 INFO memory.py L481: 'peak_ram': 21.79GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.6 | Cosine similarity 0.9988 | Peak Signal-to-Noise Ratio 72.2 | uloss 9 → 5 | iter 873/874
Quantizing model.layers.7: 17%|████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|7/40 [1:49:19<8:35:21, 937.0s/it]
2026-08-11 18:21:41 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 16 -> iter 913: 7
2026-08-11 18:21:49 INFO memory.py L481: 'peak_ram': 21.79GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.7 | Cosine similarity 0.9981 | Peak Signal-to-Noise Ratio 68.5 | uloss 16 → 7 | iter 913/914
Quantizing model.layers.8: 20%|████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|8/40 [2:03:56<8:15:46, 929.6s/it]
2026-08-11 18:37:09 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 14 -> iter 989: 8
2026-08-11 18:37:19 INFO memory.py L481: 'peak_ram': 22.07GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.8 | Cosine similarity 0.9981 | Peak Signal-to-Noise Ratio 67.3 | uloss 14 → 8 | iter 989/990
Quantizing model.layers.9: 22%|████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|9/40 [2:19:26<8:00:18, 929.6s/it]
2026-08-11 18:52:38 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 15 -> iter 974: 9
2026-08-11 18:52:48 INFO memory.py L481: 'peak_ram': 24.43GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.9 | Cosine similarity 0.9980 | Peak Signal-to-Noise Ratio 66.4 | uloss 15 → 9 | iter 974/975
Quantizing model.layers.10: 25%|███████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|10/40 [2:34:55<7:44:46, 929.6s/it]
2026-08-11 19:08:13 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 26 -> iter 841: 12
2026-08-11 19:08:23 INFO memory.py L481: 'peak_ram': 24.43GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.10 | Cosine similarity 0.9983 | Peak Signal-to-Noise Ratio 80.9 | uloss 26 → 12 | iter 841/842
Quantizing model.layers.11: 27%|███████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|11/40 [2:50:30<7:29:30, 930.0s/it]
2026-08-11 19:22:50 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 32 -> iter 994: 15
2026-08-11 19:22:59 INFO memory.py L481: 'peak_ram': 24.57GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.11 | Cosine similarity 0.9978 | Peak Signal-to-Noise Ratio 79.9 | uloss 32 → 15 | iter 994/995
Quantizing model.layers.12: 30%|███████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|12/40 [3:05:06<7:11:54, 925.5s/it]
2026-08-11 19:38:16 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 26 -> iter 945: 16
2026-08-11 19:38:27 INFO memory.py L481: 'peak_ram': 26.91GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.12 | Cosine similarity 0.9975 | Peak Signal-to-Noise Ratio 79.8 | uloss 26 → 16 | iter 945/946
Quantizing model.layers.13: 32%|███████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|13/40 [3:20:34<6:56:33, 925.7s/it]
2026-08-11 19:53:56 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 29 -> iter 977: 19
2026-08-11 19:54:06 INFO memory.py L481: 'peak_ram': 26.91GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.13 | Cosine similarity 0.9971 | Peak Signal-to-Noise Ratio 78.8 | uloss 29 → 19 | iter 977/978
Quantizing model.layers.14: 35%|███████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|14/40 [3:36:14<6:41:34, 926.7s/it]
2026-08-11 20:09:32 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 36 -> iter 975: 22
2026-08-11 20:09:42 INFO memory.py L481: 'peak_ram': 27.07GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.14 | Cosine similarity 0.9973 | Peak Signal-to-Noise Ratio 78.3 | uloss 36 → 22 | iter 975/976
Quantizing model.layers.15: 37%|███████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|15/40 [3:51:49<6:26:22, 927.3s/it]
2026-08-11 20:24:06 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 50 -> iter 673: 31
2026-08-11 20:24:15 INFO memory.py L481: 'peak_ram': 29.38GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.15 | Cosine similarity 0.9958 | Peak Signal-to-Noise Ratio 76.7 | uloss 50 → 31 | iter 673/674
Quantizing model.layers.16: 40%|███████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|16/40 [4:06:22<6:09:34, 923.9s/it]
2026-08-11 20:39:39 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 56 -> iter 885: 35
2026-08-11 20:39:49 INFO memory.py L481: 'peak_ram': 29.38GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.16 | Cosine similarity 0.9954 | Peak Signal-to-Noise Ratio 76.0 | uloss 56 → 35 | iter 885/886
Quantizing model.layers.17: 42%|███████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|17/40 [4:21:56<5:54:23, 924.5s/it]
2026-08-11 20:55:11 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 61 -> iter 466: 39
2026-08-11 20:55:21 INFO memory.py L481: 'peak_ram': 29.55GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.17 | Cosine similarity 0.9947 | Peak Signal-to-Noise Ratio 75.4 | uloss 61 → 39 | iter 466/467
Quantizing model.layers.18: 45%|███████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|18/40 [4:37:28<5:39:08, 924.9s/it]
2026-08-11 21:10:44 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 78 -> iter 838: 52
2026-08-11 21:10:54 INFO memory.py L481: 'peak_ram': 31.89GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.18 | Cosine similarity 0.9945 | Peak Signal-to-Noise Ratio 74.2 | uloss 78 → 52 | iter 838/839
Quantizing model.layers.19: 47%|███████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|19/40 [4:53:01<5:23:52, 925.4s/it]
2026-08-11 21:25:19 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 113 -> iter 840: 76
2026-08-11 21:25:28 INFO memory.py L481: 'peak_ram': 31.89GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.19 | Cosine similarity 0.9925 | Peak Signal-to-Noise Ratio 72.6 | uloss 113 → 76 | iter 840/841
Quantizing model.layers.20: 50%|███████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|20/40 [5:07:35<5:07:35, 922.8s/it]
2026-08-11 21:40:49 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 104 -> iter 686: 80
2026-08-11 21:40:59 INFO memory.py L481: 'peak_ram': 32.11GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.20 | Cosine similarity 0.9923 | Peak Signal-to-Noise Ratio 72.5 | uloss 104 → 80 | iter 686/687
Quantizing model.layers.21: 52%|██████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|21/40 [5:23:06<4:52:20, 923.2s/it]
2026-08-11 21:56:22 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 117 -> iter 823: 82
2026-08-11 21:56:33 INFO memory.py L481: 'peak_ram': 34.52GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.21 | Cosine similarity 0.9919 | Peak Signal-to-Noise Ratio 72.1 | uloss 117 → 82 | iter 823/824
Quantizing model.layers.22: 55%|██████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|22/40 [5:38:40<4:37:05, 923.6s/it]
2026-08-11 22:11:48 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 168 -> iter 791: 107
2026-08-11 22:11:59 INFO memory.py L481: 'peak_ram': 34.52GB, 'peak_vram': 28.0GB
🟢 model.language_model.layers.22 | Cosine similarity 0.9912 | Peak Signal-to-Noise Ratio 70.8 | uloss 168 → 107 | iter 791/792
Quantizing model.layers.23: 57%|██████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|23/40 [5:54:06<4:21:43, 923.7s/it]
2026-08-11 22:26:20 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 192 -> iter 585: 136
2026-08-11 22:26:29 INFO memory.py L481: 'peak_ram': 34.71GB, 'peak_vram': 28.0GB
🟠 model.language_model.layers.23 | Cosine similarity 0.9888 | Peak Signal-to-Noise Ratio 69.9 | uloss 192 → 136 | iter 585/586
Quantizing model.layers.24: 60%|██████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|24/40 [6:08:36<4:05:44, 921.5s/it]
2026-08-11 22:41:46 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 174 -> iter 681: 136
2026-08-11 22:41:57 INFO memory.py L481: 'peak_ram': 37.05GB, 'peak_vram': 28.0GB
🟠 model.language_model.layers.24 | Cosine similarity 0.9879 | Peak Signal-to-Noise Ratio 69.8 | uloss 174 → 136 | iter 681/682
Quantizing model.layers.25: 62%|██████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|25/40 [6:24:04<3:50:26, 921.8s/it]
2026-08-11 22:57:22 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 217 -> iter 414: 153
2026-08-11 22:57:32 INFO memory.py L481: 'peak_ram': 37.05GB, 'peak_vram': 28.0GB
🟠 model.language_model.layers.25 | Cosine similarity 0.9870 | Peak Signal-to-Noise Ratio 69.1 | uloss 217 → 153 | iter 414/415
Quantizing model.layers.26: 65%|██████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|26/40 [6:39:39<3:35:12, 922.3s/it]
2026-08-11 23:12:56 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 221 -> iter 581: 175
2026-08-11 23:13:06 INFO memory.py L481: 'peak_ram': 37.32GB, 'peak_vram': 28.0GB
🟠 model.language_model.layers.26 | Cosine similarity 0.9880 | Peak Signal-to-Noise Ratio 68.8 | uloss 221 → 175 | iter 581/582
Quantizing model.layers.27: 67%|██████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|27/40 [6:55:13<3:19:55, 922.7s/it]
2026-08-11 23:27:35 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 775/776 layers in the block, uloss iter 0: 284 -> iter 974: 215
2026-08-11 23:27:44 INFO memory.py L481: 'peak_ram': 39.71GB, 'peak_vram': 28.0GB
🟠 model.language_model.layers.27 | Cosine similarity 0.9859 | Peak Signal-to-Noise Ratio 67.9 | uloss 284 → 215 | iter 974/975
Quantizing model.layers.28: 70%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░|28/40 [7:09:52<3:04:13, 921.1s/it]
2026-08-11 23:43:11 INFO quantizer.py L321: Unquantized layers: ['mlp.shared_expert_gate']
quantized 776/777 layers in the block, uloss iter 0: 302 -> iter 690: 231
2026-08-11 23:43:22 INFO memory.py L481: 'peak_ram': 39.71GB, 'peak_vram': 28.0GB

I think there may be an issue with the checkpoint resume. Kimi seemed to think there could be an issue with the input layers being corrupted or not passing properly to the new layer during the resume. I would be careful if this happens to anyone else to verify the numbers if you do a resume - I currently recommend doing a full clear cache or wipe and starting over if your run gets interrupted.

6

Here is another 35B-A3B class model called BigBang-V1 with strong benchmark claims. I gave it the spark autoround opencode treatment and I believe the quality is solid. I haven’t done much testing on it but it came in behind KAT coder on tool eval benchmark with 82.

bigbang_main_figure
bigbang_main_figure1440×1353 179 KB