# New 2.5x Faster Qwen3.6 NVFP4 Unsloth quants

**URL:** <https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484>\
**Category:** DGX Spark / GB10\
**Created:** [July 10, 2026, 3:10pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484 "2026-07-10T15:10:30Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![emX0r](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@emX0r](https://forums.developer.nvidia.com/u/emX0r)\
**Post date:** [July 10, 2026, 3:10pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/1 "2026-07-10T15:10:30Z")

</div>

> <https://x.com/UnslothAI/status/2075566124687892597>

from what they say…  
"Hey guys, we’re excited to release new Qwen3.6 NVFP4 quants that run 2.5× faster on your GPU.

Qwen3.6-27B NVFP4 runs on 24GB VRAM. 35B-A3B can hit 17,561 tok/s (B200).

We also improved accuracy, tool calling, agent use, and looping.

Guide + Benchmarks: [Qwen3.6 - How to Run Locally | Unsloth Documentation](https://unsloth.ai/docs/models/qwen3.6#nvfp4)

Qwen3.6 NVFP4: [NVFP4 - a unsloth Collection](https://huggingface.co/collections/unsloth/nvfp4)"

---

<div class="post-metadata">

**Author:** ![blainesworld](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/blainesworld/32/472655_2.png) [@blainesworld](https://forums.developer.nvidia.com/u/blainesworld)\
**Post date:** [July 10, 2026, 3:38pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/2 "2026-07-10T15:38:28Z")

</div>

Anyone try it out yet?

---

<div class="post-metadata">

**Author:** ![cosinus](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/cosinus/32/454111_2.png) [@cosinus](https://forums.developer.nvidia.com/u/cosinus)\
**Post date:** [July 10, 2026, 4:12pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/3 "2026-07-10T16:12:56Z")

</div>

> Works on a 32GB VRAM GPU. **Benchmarks on 1xB200 128 concurrency.**

Well. Benchmarked on B200. Cool. I don’t expect much on GPUs below B200… “Consumer Blackwell” or did I miss some breakthroughs on sm121 and NVFP4?

also [Reddit - Please wait for verification](https://www.reddit.com/r/Vllm/comments/1uki6f8/nvfp4_still_isnt_faster_than_fp8_on_blackwell/)

---

<div class="post-metadata">

**Author:** ![nshotton](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@nshotton](https://forums.developer.nvidia.com/u/nshotton)\
**Post date:** [July 10, 2026, 4:14pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/4 "2026-07-10T16:14:04Z")

</div>

It looks promising, im going to run some tests on it today and see how it performs.

---

<div class="post-metadata">

**Author:** ![nshotton](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@nshotton](https://forums.developer.nvidia.com/u/nshotton)\
**Post date:** [July 10, 2026, 5:18pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/5 "2026-07-10T17:18:58Z")

</div>

| model | test | t/s | peak t/s | ttfr (ms) | est\_ppt (ms) | e2e\_ttft (ms) |

|:----------------|----------------:|-----------------:|--------------:|----------------:|----------------:|----------------:|

| qwen3.6-35b-a3b | pp2048 | 5356.82 ± 614.93 | | 438.90 ± 47.86 | 387.99 ± 47.86 | 438.90 ± 47.86 |

| qwen3.6-35b-a3b | tg32 | 84.60 ± 7.79 | 87.33 ± 8.04 | | | |

| qwen3.6-35b-a3b | ctx\_pp @ d4096 | 6255.12 ± 79.15 | | 706.10 ± 8.27 | 655.19 ± 8.27 | 706.10 ± 8.27 |

| qwen3.6-35b-a3b | ctx\_tg @ d4096 | 100.66 ± 4.84 | 103.91 ± 4.99 | | | |

| qwen3.6-35b-a3b | pp2048 @ d4096 | 2097.95 ± 9.43 | | 1027.12 ± 4.40 | 976.21 ± 4.40 | 1027.12 ± 4.40 |

| qwen3.6-35b-a3b | tg32 @ d4096 | 89.00 ± 3.15 | 91.87 ± 3.25 | | | |

| qwen3.6-35b-a3b | ctx\_pp @ d8192 | 6109.01 ± 15.43 | | 1392.27 ± 3.38 | 1341.36 ± 3.38 | 1392.27 ± 3.38 |

| qwen3.6-35b-a3b | ctx\_tg @ d8192 | 94.29 ± 0.40 | 97.34 ± 0.42 | | | |

| qwen3.6-35b-a3b | pp2048 @ d8192 | 1207.57 ± 1.34 | | 1746.88 ± 1.88 | 1695.97 ± 1.88 | 1746.88 ± 1.88 |

| qwen3.6-35b-a3b | tg32 @ d8192 | 92.03 ± 3.49 | 94.99 ± 3.60 | | | |

| qwen3.6-35b-a3b | ctx\_pp @ d16384 | 5660.44 ± 8.19 | | 2945.68 ± 4.27 | 2894.78 ± 4.27 | 2945.68 ± 4.27 |

| qwen3.6-35b-a3b | ctx\_tg @ d16384 | 89.69 ± 7.43 | 92.58 ± 7.67 | | | |

| qwen3.6-35b-a3b | pp2048 @ d16384 | 618.55 ± 1.95 | | 3361.93 ± 10.44 | 3311.02 ± 10.44 | 3362.49 ± 10.89 |

| qwen3.6-35b-a3b | tg32 @ d16384 | 98.04 ± 4.61 | 101.20 ± 4.76 | | | |

| qwen3.6-35b-a3b | ctx\_pp @ d32768 | 4848.47 ± 8.69 | | 6809.77 ± 12.13 | 6758.86 ± 12.13 | 6812.46 ± 11.95 |

| qwen3.6-35b-a3b | ctx\_tg @ d32768 | 100.05 ± 4.81 | 103.27 ± 4.96 | | | |

| qwen3.6-35b-a3b | pp2048 @ d32768 | 280.56 ± 0.54 | | 7350.50 ± 14.18 | 7299.59 ± 14.18 | 7353.61 ± 14.25 |

| qwen3.6-35b-a3b | tg32 @ d32768 | 91.67 ± 4.32 | 94.62 ± 4.46 | | | |

---

<div class="post-metadata">

**Author:** ![hedelyuk.alexandr](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@hedelyuk.alexandr](https://forums.developer.nvidia.com/u/hedelyuk.alexandr)\
**Post date:** [July 10, 2026, 5:24pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/6 "2026-07-10T17:24:04Z")

</div>

unsloth/Qwen3.6-35B-A3B-NVFP4 vs nvidia/Qwen3.6-35B-A3B-NVFP4

=== short\_ok ===  
NVIDIA decode tok/s: 107.16  
Unsloth decode tok/s: 92.91  
Unsloth delta: -13.3%  
NVIDIA TTFT p50: 0.072s  
Unsloth TTFT p50: 0.085s  
NVIDIA total avg: 1.11s  
Unsloth total avg: 2.20s

=== coding\_small ===  
NVIDIA decode tok/s: 112.30  
Unsloth decode tok/s: 95.71  
Unsloth delta: -14.8%  
NVIDIA TTFT p50: 0.094s  
Unsloth TTFT p50: 0.104s  
NVIDIA total avg: 9.94s  
Unsloth total avg: 11.91s

=== reasoning\_medium ===  
NVIDIA decode tok/s: 93.20  
Unsloth decode tok/s: 77.29  
Unsloth delta: -17.1%  
NVIDIA TTFT p50: 0.095s  
Unsloth TTFT p50: 0.110s  
NVIDIA total avg: 22.07s  
Unsloth total avg: 26.72s

=== coding\_agentic ===  
NVIDIA decode tok/s: 100.82  
Unsloth decode tok/s: 84.64  
Unsloth delta: -16.1%  
NVIDIA TTFT p50: 0.101s  
Unsloth TTFT p50: 0.120s  
NVIDIA total avg: 20.42s  
Unsloth total avg: 24.46s

=== average decode tok/s ===  
NVIDIA: 103.37  
Unsloth: 87.64  
Delta: -15.2%

---

<div class="post-metadata">

**Author:** ![J-R](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@J-R](https://forums.developer.nvidia.com/u/J-R)\
**Post date:** [July 10, 2026, 5:24pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/7 "2026-07-10T17:24:19Z")

</div>

Just tested 35B-A3B. Comparing it (unsloth) to nvidia’s (nv) NVFP4 version:  
vLLM 0.24, MTP=2  
concurrency=1, nv~90tps, unsloth~75tps(GPU@25W)  
concurrency=4, nv~220tps, unsloth~195tps(GPU@28W)  
concurrency=8, nv~310tps, unsloth~290tps(GPU@29W)  
concurrency=16, nv~420tps, unsloth~410tps(GPU@32W)

---

<div class="post-metadata">

**Author:** ![jbourny](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jbourny](https://forums.developer.nvidia.com/u/jbourny)\
**Post date:** [July 11, 2026, 11:49am UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/9 "2026-07-11T11:49:04Z")

</div>

> [@emX0r](#):
>
> [Unsloth AI on X: "We’re releasing new Qwen3.6 quants that run 2.5× faster on your GPU. Qwen3.6-27B NVFP4 runs on 24GB VRAM. 35B-A3B can hit 17,561 tok/s (B200). We also improved accuracy, tool calling, agent use, and looping. Guide: https://t.co/EEQIlFrR0c Qwen3.6 NVFP4: https://t.co/RWflncpLPJ https://t.co/MSlxEWm4a5" / X](https://x.com/UnslothAI/status/2075566124687892597)

I can’t test it doesn’t work, each time it says flashinfer\_b12x is not available, I don’t know how you did…  
`(EngineCore pid=1690683) ValueError: --linear-backend=flashinfer_b12x was requested but no 'flashinfer_b12x' kernel exists for this layer type.`

```auto
python -c "

import torch; from vllm.utils.flashinfer import has_flashinfer_b12x_gemm as g, has_flashinfer_b12x_moe as m

cap = torch.cuda.get_device_capability(); print('cap', cap, '| b12x gemm', g(), '| b12x moe', m()); assert cap[0] == 12 and g() and m(), 'b12x unavailable: serving would degrade to marlin W4A16'"

>>> cap (12, 1) | b12x gemm True | b12x moe True

```

```auto
source myenv/bin/activate
export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
--host 0.0.0.0 \
--port 8001 \
--max-model-len 180000 \
--max-num-batched-tokens 32768 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.4 \
--moe-backend flashinfer_b12x \
--linear-backend flashinfer_b12x

```

And when I try with spark-vllm-docker without b12x cause I’ve the same issue, it’s worse than Nvidia Qwen3.6 35b

---

<div class="post-metadata">

**Author:** ![JW2026](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jw2026/32/523471_2.png) [@JW2026](https://forums.developer.nvidia.com/u/JW2026)\
**Post date:** [July 11, 2026, 12:09pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/10 "2026-07-11T12:09:07Z")

</div>

> [@jbourny](#):
>
> I can’t test it doesn’t work, each time it says flashinfer\_b12x is not available, I don’t know how you did…

According to Germini -

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/2/9/3/29371eef0ce64e4257788604100f5e3bbebb2e0f.png)

---

<div class="post-metadata">

**Author:** ![emX0r](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@emX0r](https://forums.developer.nvidia.com/u/emX0r)\
**Post date:** [July 11, 2026, 12:42pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/11 "2026-07-11T12:42:14Z")

</div>

I got it working using spark-vllm-docker using following recipe:

```auto
recipe_version: "1"
name: Qwen3.6-35B-A3B-NVFP4-unsloth
description: vLLM serving unsloth/Qwen3.6-35B-A3B-NVFP4 with Marlin MoE backend on a single Spark

# HuggingFace model to download (used by --setup / --download-only)
model: unsloth/Qwen3.6-35B-A3B-NVFP4

# Container image to use (default runner image)
container: vllm-node

# Fits on one Spark. Not solo_only, so you can -tp 2 across a pair of Sparks
# for more KV-cache headroom / concurrency.
cluster_only: false

# Default settings (override via CLI, e.g. --port, --gpu-mem, --max-model-len, --tp)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.8
  max_model_len: 262144
  max_num_seqs: 8
  max_num_batched_tokens: 8192
  served_model_name: qwen3.6-35b-a3b

# Environment variables
# Marlin is the working NVFP4 MoE path on sm121 (GB10) until FlashInfer NVFP4 is
# fully supported; the atomic-add flag improves its throughput.
env:
  VLLM_MARLIN_USE_ATOMIC_ADD: 1

# The vLLM serve command template.
# Notes:
# - --moe-backend marlin is required here (this IS a MoE model), unlike the
# dense 27B recipe.
# - MTP uses the checkpoint's own head as a draft for a free decode speedup.
# If your vLLM build rejects "method":"mtp", the Qwen card's alternative is
# '{{"method":"qwen3_next_mtp","num_speculative_tokens":2}}'. If MTP is
# unstable on a given build, delete that whole line (see the repo's
# qwen3.6-35b-a3b-nvfp4-no-mtp.yaml for the no-MTP variant).
# - Coding-only / no images: append `-- --language-model-only` to skip the
# vision encoder and free that memory for extra KV cache.
# - OOM at load: lower --max-model-len first (e.g. 131072; keep >=128K to
# preserve thinking quality), then --gpu-memory-utilization, then
# --max-num-seqs. For >262144 context use YaRN via --hf-overrides +
# VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (see model card).
command: |
  vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 \
    --served-model-name {served_model_name} \
    --host {host} \
    --port {port} \
    --tensor-parallel-size {tensor_parallel} \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --attention-backend flashinfer \
    --moe-backend marlin \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --enable-chunked-prefill \
    --async-scheduling \
    --enable-prefix-caching \
    --speculative-config '{{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}}' \
    --load-format fastsafetensors \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
    --distributed-executor-backend ray

```

---

<div class="post-metadata">

**Author:** ![JW2026](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jw2026/32/523471_2.png) [@JW2026](https://forums.developer.nvidia.com/u/JW2026)\
**Post date:** [July 11, 2026, 12:48pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/12 "2026-07-11T12:48:16Z")

</div>

> [@emX0r](#):
>
> I got it working using spark-vllm-docker using following recipe:
> 
> ```auto
> 
> ```

Reason he asked is because Unsloth cliams that the bx121 flags are necessary for optimum performences but their recipe appears to be based on vLLM 0.24 and has specific b121 flags. I tested and the differences are less than 3%, within margin of errors.

---

<div class="post-metadata">

**Author:** ![JW2026](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jw2026/32/523471_2.png) [@JW2026](https://forums.developer.nvidia.com/u/JW2026)\
**Post date:** [July 11, 2026, 4:36pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/13 "2026-07-11T16:36:36Z")

</div>

![image](https://global.discourse-cdn.com/nvidia/original/4X/7/b/9/7b92b5a14e8052bf3a23c2af4930e0de58d65abe.png)

Benchmarks for Unsloth vs current quant of QWEN 3.6 35B MoE.

\*\* aura-dv2 is basically just a Qwen 3.6 35B 5.1 bpp quant using PrismaAura. The bench scores are not a reflection of the quality of PrismaAURA as I was doing testings that might impact the actual performance of the quant.

> **[JasonW2025/Qwen3.6-35B-A3B-HybridQuant-NVFP4 · Hugging Face](https://huggingface.co/JasonW2025/Qwen3.6-35B-A3B-HybridQuant-NVFP4)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![TheAwakenOne](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/theawakenone/32/498282_2.png) [@TheAwakenOne](https://forums.developer.nvidia.com/u/TheAwakenOne)\
**Post date:** [July 11, 2026, 8:02pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/14 "2026-07-11T20:02:25Z")

</div>

I just got unsloth/Qwen3.6-35B-A3B-NVFP4-Fast running from 75 tok/s to 97 tok/s but still missing the --moe-backend flashinfer\_b12x and --linear-backend flashinfer\_b12x. I also need to change MTP num\_speculative\_tokens from 3 to 2. Will mess around with it a bit more.

recipe\_version: “1”  
name: Qwen3.6-35B-A3B-NVFP4-Fast-Unsloth-Marlin-MTP-Solo  
description: vLLM serving unsloth/Qwen3.6-35B-A3B-NVFP4-Fast on one DGX Spark node with Marlin MoE backend and MTP speculative decoding (drafter MoE pinned to triton)

model: unsloth/Qwen3.6-35B-A3B-NVFP4-Fast

container: vllm-node

defaults:  
port: 8000  
host: 0.0.0.0  
tensor\_parallel: 1  
gpu\_memory\_utilization: 0.6  
max\_model\_len: 262144  
max\_num\_seqs: 4  
max\_num\_batched\_tokens: 16384

env:  
VLLM\_MARLIN\_USE\_ATOMIC\_ADD: 1  
CUTE\_DSL\_ARCH: sm\_121a

command: |  
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast  
–host {host}  
–port {port}  
–tensor-parallel-size {tensor\_parallel}  
–trust-remote-code  
–kv-cache-dtype fp8  
–attention-backend flashinfer  
–moe-backend marlin  
–speculative-config ‘{“method”:“mtp”,“num\_speculative\_tokens”:3,“moe\_backend”:“triton”}’  
–gpu-memory-utilization {gpu\_memory\_utilization}  
–max-model-len {max\_model\_len}  
–max-num-seqs {max\_num\_seqs}  
–max-num-batched-tokens {max\_num\_batched\_tokens}  
–enable-chunked-prefill  
–async-scheduling  
–enable-prefix-caching  
–load-format fastsafetensors  
–reasoning-parser qwen3  
–tool-call-parser qwen3\_xml  
–enable-auto-tool-choice

---

<div class="post-metadata">

**Author:** ![posetoimpress](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@posetoimpress](https://forums.developer.nvidia.com/u/posetoimpress)\
**Post date:** [July 12, 2026, 7:59pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/15 "2026-07-12T19:59:11Z")

</div>

> [@emX0r](#):
>
> ```auto
> --speculative-config '{{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}}' \
> 
> ```

Thank you for posting the recipe! I tried this but got an error for speculative-config so changed

```
*--speculative-config '{{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}}' \\

```

to\*

```
*--speculative-config '{{"method":"mtp","num_speculative_tokens":3}}' \\*

```

Now it gets past the error but I get

_Download complete: : 67.4kB [00:20, 213kB/s]Error: inference server did not become ready_

I run sparkrun benchmark against existing recipes to make sure my setup is OK. See below:

Waiting for inference server on 127.0.0.1:8000…

Note that this could take ~5 minutes!

Waiting for model to finish loading ([http://127.0.0.1:8000/v1/models](http://127.0.0.1:8000/v1/models))…

Inference server ready.

Step 2/3: Running benchmark (llama-benchy)…

[11:08:19] Task **1** / **1** done **(10.2s)** progress\_ui.py:158

**0**** 1 ****4002.1**** 52.6 ****514.8**** 3**

**[1/1] d=0 c=1** ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 100.0% 0:00:00 0:00:10

┏━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━┓

┃ **depth** ┃ **conc** ┃ **pp t/s** ┃ **tg t/s** ┃ **ttfr ms** ┃ **runs** ┃

┡━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━┩

│ 0 │ 1 │ 4002.1 │ 52.6 │ 514.8 │ 3 │

└───────┴──────┴────────┴────────┴─────────┴──────┘

┏━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━┓

┃ **depth** ┃ **conc** ┃ **pp t/s** ┃ **tg t/s** ┃ **ttfr ms** ┃ **runs** ┃

┡━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━┩

│ 0 │ 1 │ 4002.1 │ 52.6 │ 514.8 │ 3 │

└───────┴──────┴────────┴────────┴─────────┴──────┘

My first time trying a new recipe on the spark (Asus GX10) so assume I am doing something wrong.

I created a yaml file under ~/.cache/sparkrun/registries/official/official-recipes/qwen3.6/vllm/ and copied the above into that file.

A few recipes that are listed under “sparkrun list” also do not run on my system. How do I debug it?

thanks for any help!

---

<div class="post-metadata">

**Author:** ![jbourny](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jbourny](https://forums.developer.nvidia.com/u/jbourny)\
**Post date:** [July 13, 2026, 10:03am UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/16 "2026-07-13T10:03:24Z")

</div>

Yes it works but it’s slower than Nvidia :

```bash
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────── 🔮 Speculative Decoding Benchmark ─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ nvidia/Qwen3.6-35B-A3B-NVFP4 │
│ tg=128 depth=[0, 16384, 32768] prompts=['filler', 'code', 'structured'] method=auto │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Prometheus /metrics acceptance-rate counters are server-wide aggregates. If other models are serving concurrent traffic on this endpoint, per-request acceptance rate measurements will be inaccurate. For clean measurements: use a single-model server with no concurrent load.

  ✓ filler @ d0 88.0 eff t/s 87.4 stream t/s α=78.9% waste=21% τ=2.4 win=3
  ✓ code @ d0 117.6 eff t/s 116.7 stream t/s α=92.2% waste=8% τ=2.8 win=3
  ✓ structured @ d0 103.0 eff t/s 102.2 stream t/s α=76.9% waste=23% τ=2.3 win=3
  ✓ filler @ d16384 69.6 eff t/s 69.1 stream t/s α=78.1% waste=22% τ=2.3 win=3
  ✓ code @ d16384 111.6 eff t/s 110.8 stream t/s α=83.8% waste=16% τ=2.5 win=3
  ✓ structured @ d16384 100.0 eff t/s 99.3 stream t/s α=72.4% waste=28% τ=2.2 win=3
  ✓ filler @ d32768 62.9 eff t/s 62.4 stream t/s α=62.9% waste=37% τ=1.9 win=3
  ✓ code @ d32768 111.3 eff t/s 110.5 stream t/s α=83.8% waste=16% τ=2.5 win=3
  ✓ structured @ d32768 99.8 eff t/s 99.0 stream t/s α=72.4% waste=28% τ=2.2 win=3

                                  Speculative Decoding Results                                  
┏━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃ Prompt ┃ Depth ┃ Eff t/s ┃ α % ┃ Waste ┃ τ len ┃ Win ┃ Draft t/s ┃ TTFT ms ┃ Total ms ┃
┡━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│ filler │ 0 │ 88.0 │ 78.9% │ 21% │ 2.4 │ 3 │ 78.4 │ 6 │ 1,459 │
│ code │ 0 │ 117.6 │ 92.2% │ 8% │ 2.8 │ 3 │ 93.7 │ 3 │ 1,091 │
│ structured │ 0 │ 103.0 │ 76.9% │ 23% │ 2.3 │ 3 │ 94.2 │ 3 │ 1,245 │
│ filler │ 16K │ 69.6 │ 78.1% │ 22% │ 2.3 │ 3 │ 62.0 │ 22 │ 1,862 │
│ code │ 16K │ 111.6 │ 83.8% │ 16% │ 2.5 │ 3 │ 96.8 │ 2 │ 1,149 │
│ structured │ 16K │ 100.0 │ 72.4% │ 28% │ 2.2 │ 3 │ 96.1 │ 3 │ 1,283 │
│ filler │ 32K │ 62.9 │ 62.9% │ 37% │ 1.9 │ 3 │ 64.9 │ 42 │ 2,076 │
│ code │ 32K │ 111.3 │ 83.8% │ 16% │ 2.5 │ 3 │ 96.5 │ 2 │ 1,152 │
│ structured │ 32K │ 99.8 │ 72.4% │ 28% │ 2.2 │ 3 │ 95.9 │ 4 │ 1,288 │
└────────────┴───────┴─────────┴────────┴───────┴───────┴─────┴───────────┴─────────┴──────────┘

  Highest acceptance: code (92.2%) Lowest: filler (62.9%)
  Draft window: 2.3/3 positions used (78% utilization) Avg waste: 22%

```

My recipe for spark-vllm-docker ( vLLM **0.23**.1rc1.dev1043+ga4b4b5787.d20260711) :

```yaml

recipe_version: '1'
name: Qwen3.6 35B
model: nvidia/Qwen3.6-35B-A3B-NVFP4
container: vllm-node
cluster_only: false
solo_only: false

mods:
  - mods/fix-qwen3.6-enhanced-chat-template

defaults:
  host: 0.0.0.0
  port: 8001
  max_model_len: 120000
  max_num_batched_tokens: 16384
  max_num_seqs: 2
  tensor_parallel: 2
  gpu_memory_utilization: 0.3 #2.24x
  num_speculative_tokens: 3
  tokenizer_mode: auto
  chat_template: qwen3.6-enhanced.jinja
  max_lora_rank: 32

env:
  VLLM_MARLIN_USE_ATOMIC_ADD: 1
command: |
  vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
    --host {host} \
    --port {port} \
    --tensor-parallel-size {tensor_parallel} \
    --max-model-len {max_model_len} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --max-num-seqs {max_num_seqs} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --dtype auto \
    --attention-backend flashinfer \
    --moe-backend marlin \
    --tokenizer-mode {tokenizer_mode} \
    --load-format instanttensor \
    --async-scheduling \
    --enable-chunked-prefill \
    --chat-template {chat_template} \
    --enable-prefix-caching \
    --trust-remote-code \
    --quantization compressed-tensors \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --kv-cache-dtype fp8_e4m3 \
    --default-chat-template-kwargs '{{"preserve_thinking":true}}' \
    --speculative-config '{{"method":"mtp","num_speculative_tokens":{num_speculative_tokens},"moe_backend":"triton"}}' \
    --override-generation-config '{{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05,"max_tokens":80000}}'

```

---

<div class="post-metadata">

**Author:** ![azampatti](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/azampatti/32/503643_2.png) [@azampatti](https://forums.developer.nvidia.com/u/azampatti)\
**Post date:** [July 13, 2026, 3:15pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/17 "2026-07-13T15:15:50Z")

</div>

I agree, I just tried three different configs and looks about ~10% slower than nVidia’s at least. I’ll run my usual set of benchmarks to evaluate quality as well.

---

<div class="post-metadata">

**Author:** ![Digital\_David](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/digital_david/32/495104_2.png) [@Digital\_David](https://forums.developer.nvidia.com/u/Digital_David)\
**Post date:** [July 13, 2026, 4:49pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/18 "2026-07-13T16:49:10Z")

</div>

Can someone run the tool-eval-bench --base-url [http://0.0.0.0:8000](http://0.0.0.0:8000) --seed 42 --perf so we can see how it handles tools?

---

<div class="post-metadata">

**Author:** ![azampatti](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/azampatti/32/503643_2.png) [@azampatti](https://forums.developer.nvidia.com/u/azampatti)\
**Post date:** [July 13, 2026, 4:52pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/19 "2026-07-13T16:52:47Z")

</div>

Quality looks about the same as the regular nvidia/NVFP4 one but with more variance between runs. I would call it equally good.

---

<div class="post-metadata">

**Author:** ![azampatti](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/azampatti/32/503643_2.png) [@azampatti](https://forums.developer.nvidia.com/u/azampatti)\
**Post date:** [July 13, 2026, 5:08pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/20 "2026-07-13T17:08:10Z")

</div>

It’s about the same as nvidia’s , like I said, here’s the full `tool-eval-bench --seed 42 --perf` output:

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/5/6/d/56d61fdb128a2b6eda0ffba7f538cb2d4dfb49a7.png)

(trimmed the individual task output for convenience)

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/f/f/f/fffeac81c95ac1a4e42f3458a6736af5768e6613.png)

---

<div class="post-metadata">

**Author:** ![robert287](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@robert287](https://forums.developer.nvidia.com/u/robert287)\
**Post date:** [July 13, 2026, 6:55pm UTC](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484/21 "2026-07-13T18:55:02Z")

</div>

Correct me if I am wrong but Unsloth’s “performance” boost for Qwen models was just done by bypassing the NVFP4 bottleneck on weights by using W4A16

[Next page](https://forums.developer.nvidia.com/t/new-2-5x-faster-qwen3-6-nvfp4-unsloth-quants/376484.md?page=2)
