from what they say…
"Hey guys, we’re excited to release new Qwen3.6 NVFP4 quants that run 2.5× faster on your GPU.
Qwen3.6-27B NVFP4 runs on 24GB VRAM. 35B-A3B can hit 17,561 tok/s (B200).
We also improved accuracy, tool calling, agent use, and looping.
Guide + Benchmarks: Qwen3.6 - How to Run Locally | Unsloth Documentation
Qwen3.6 NVFP4: NVFP4 - a unsloth Collection"
Anyone try it out yet?
Works on a 32GB VRAM GPU. Benchmarks on 1xB200 128 concurrency.
Well. Benchmarked on B200. Cool. I don’t expect much on GPUs below B200… “Consumer Blackwell” or did I miss some breakthroughs on sm121 and NVFP4?
It looks promising, im going to run some tests on it today and see how it performs.
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:----------------|----------------:|-----------------:|--------------:|----------------:|----------------:|----------------:|
| qwen3.6-35b-a3b | pp2048 | 5356.82 ± 614.93 | | 438.90 ± 47.86 | 387.99 ± 47.86 | 438.90 ± 47.86 |
| qwen3.6-35b-a3b | tg32 | 84.60 ± 7.79 | 87.33 ± 8.04 | | | |
| qwen3.6-35b-a3b | ctx_pp @ d4096 | 6255.12 ± 79.15 | | 706.10 ± 8.27 | 655.19 ± 8.27 | 706.10 ± 8.27 |
| qwen3.6-35b-a3b | ctx_tg @ d4096 | 100.66 ± 4.84 | 103.91 ± 4.99 | | | |
| qwen3.6-35b-a3b | pp2048 @ d4096 | 2097.95 ± 9.43 | | 1027.12 ± 4.40 | 976.21 ± 4.40 | 1027.12 ± 4.40 |
| qwen3.6-35b-a3b | tg32 @ d4096 | 89.00 ± 3.15 | 91.87 ± 3.25 | | | |
| qwen3.6-35b-a3b | ctx_pp @ d8192 | 6109.01 ± 15.43 | | 1392.27 ± 3.38 | 1341.36 ± 3.38 | 1392.27 ± 3.38 |
| qwen3.6-35b-a3b | ctx_tg @ d8192 | 94.29 ± 0.40 | 97.34 ± 0.42 | | | |
| qwen3.6-35b-a3b | pp2048 @ d8192 | 1207.57 ± 1.34 | | 1746.88 ± 1.88 | 1695.97 ± 1.88 | 1746.88 ± 1.88 |
| qwen3.6-35b-a3b | tg32 @ d8192 | 92.03 ± 3.49 | 94.99 ± 3.60 | | | |
| qwen3.6-35b-a3b | ctx_pp @ d16384 | 5660.44 ± 8.19 | | 2945.68 ± 4.27 | 2894.78 ± 4.27 | 2945.68 ± 4.27 |
| qwen3.6-35b-a3b | ctx_tg @ d16384 | 89.69 ± 7.43 | 92.58 ± 7.67 | | | |
| qwen3.6-35b-a3b | pp2048 @ d16384 | 618.55 ± 1.95 | | 3361.93 ± 10.44 | 3311.02 ± 10.44 | 3362.49 ± 10.89 |
| qwen3.6-35b-a3b | tg32 @ d16384 | 98.04 ± 4.61 | 101.20 ± 4.76 | | | |
| qwen3.6-35b-a3b | ctx_pp @ d32768 | 4848.47 ± 8.69 | | 6809.77 ± 12.13 | 6758.86 ± 12.13 | 6812.46 ± 11.95 |
| qwen3.6-35b-a3b | ctx_tg @ d32768 | 100.05 ± 4.81 | 103.27 ± 4.96 | | | |
| qwen3.6-35b-a3b | pp2048 @ d32768 | 280.56 ± 0.54 | | 7350.50 ± 14.18 | 7299.59 ± 14.18 | 7353.61 ± 14.25 |
| qwen3.6-35b-a3b | tg32 @ d32768 | 91.67 ± 4.32 | 94.62 ± 4.46 | | | |
unsloth/Qwen3.6-35B-A3B-NVFP4 vs nvidia/Qwen3.6-35B-A3B-NVFP4
=== short_ok ===
NVIDIA decode tok/s: 107.16
Unsloth decode tok/s: 92.91
Unsloth delta: -13.3%
NVIDIA TTFT p50: 0.072s
Unsloth TTFT p50: 0.085s
NVIDIA total avg: 1.11s
Unsloth total avg: 2.20s
=== coding_small ===
NVIDIA decode tok/s: 112.30
Unsloth decode tok/s: 95.71
Unsloth delta: -14.8%
NVIDIA TTFT p50: 0.094s
Unsloth TTFT p50: 0.104s
NVIDIA total avg: 9.94s
Unsloth total avg: 11.91s
=== reasoning_medium ===
NVIDIA decode tok/s: 93.20
Unsloth decode tok/s: 77.29
Unsloth delta: -17.1%
NVIDIA TTFT p50: 0.095s
Unsloth TTFT p50: 0.110s
NVIDIA total avg: 22.07s
Unsloth total avg: 26.72s
=== coding_agentic ===
NVIDIA decode tok/s: 100.82
Unsloth decode tok/s: 84.64
Unsloth delta: -16.1%
NVIDIA TTFT p50: 0.101s
Unsloth TTFT p50: 0.120s
NVIDIA total avg: 20.42s
Unsloth total avg: 24.46s
=== average decode tok/s ===
NVIDIA: 103.37
Unsloth: 87.64
Delta: -15.2%
Just tested 35B-A3B. Comparing it (unsloth) to nvidia’s (nv) NVFP4 version:
vLLM 0.24, MTP=2
concurrency=1, nv~90tps, unsloth~75tps(GPU@25W)
concurrency=4, nv~220tps, unsloth~195tps(GPU@28W)
concurrency=8, nv~310tps, unsloth~290tps(GPU@29W)
concurrency=16, nv~420tps, unsloth~410tps(GPU@32W)
I can’t test it doesn’t work, each time it says flashinfer_b12x is not available, I don’t know how you did…
(EngineCore pid=1690683) ValueError: --linear-backend=flashinfer_b12x was requested but no 'flashinfer_b12x' kernel exists for this layer type.
python -c "
import torch; from vllm.utils.flashinfer import has_flashinfer_b12x_gemm as g, has_flashinfer_b12x_moe as m
cap = torch.cuda.get_device_capability(); print('cap', cap, '| b12x gemm', g(), '| b12x moe', m()); assert cap[0] == 12 and g() and m(), 'b12x unavailable: serving would degrade to marlin W4A16'"
>>> cap (12, 1) | b12x gemm True | b12x moe True
source myenv/bin/activate
export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
--host 0.0.0.0 \
--port 8001 \
--max-model-len 180000 \
--max-num-batched-tokens 32768 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.4 \
--moe-backend flashinfer_b12x \
--linear-backend flashinfer_b12x
And when I try with spark-vllm-docker without b12x cause I’ve the same issue, it’s worse than Nvidia Qwen3.6 35b
I got it working using spark-vllm-docker using following recipe:
recipe_version: "1"
name: Qwen3.6-35B-A3B-NVFP4-unsloth
description: vLLM serving unsloth/Qwen3.6-35B-A3B-NVFP4 with Marlin MoE backend on a single Spark
# HuggingFace model to download (used by --setup / --download-only)
model: unsloth/Qwen3.6-35B-A3B-NVFP4
# Container image to use (default runner image)
container: vllm-node
# Fits on one Spark. Not solo_only, so you can -tp 2 across a pair of Sparks
# for more KV-cache headroom / concurrency.
cluster_only: false
# Default settings (override via CLI, e.g. --port, --gpu-mem, --max-model-len, --tp)
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.8
max_model_len: 262144
max_num_seqs: 8
max_num_batched_tokens: 8192
served_model_name: qwen3.6-35b-a3b
# Environment variables
# Marlin is the working NVFP4 MoE path on sm121 (GB10) until FlashInfer NVFP4 is
# fully supported; the atomic-add flag improves its throughput.
env:
VLLM_MARLIN_USE_ATOMIC_ADD: 1
# The vLLM serve command template.
# Notes:
# - --moe-backend marlin is required here (this IS a MoE model), unlike the
# dense 27B recipe.
# - MTP uses the checkpoint's own head as a draft for a free decode speedup.
# If your vLLM build rejects "method":"mtp", the Qwen card's alternative is
# '{{"method":"qwen3_next_mtp","num_speculative_tokens":2}}'. If MTP is
# unstable on a given build, delete that whole line (see the repo's
# qwen3.6-35b-a3b-nvfp4-no-mtp.yaml for the no-MTP variant).
# - Coding-only / no images: append `-- --language-model-only` to skip the
# vision encoder and free that memory for extra KV cache.
# - OOM at load: lower --max-model-len first (e.g. 131072; keep >=128K to
# preserve thinking quality), then --gpu-memory-utilization, then
# --max-num-seqs. For >262144 context use YaRN via --hf-overrides +
# VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (see model card).
command: |
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 \
--served-model-name {served_model_name} \
--host {host} \
--port {port} \
--tensor-parallel-size {tensor_parallel} \
--trust-remote-code \
--kv-cache-dtype fp8 \
--attention-backend flashinfer \
--moe-backend marlin \
--gpu-memory-utilization {gpu_memory_utilization} \
--max-model-len {max_model_len} \
--max-num-seqs {max_num_seqs} \
--max-num-batched-tokens {max_num_batched_tokens} \
--enable-chunked-prefill \
--async-scheduling \
--enable-prefix-caching \
--speculative-config '{{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}}' \
--load-format fastsafetensors \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice \
--distributed-executor-backend ray
Reason he asked is because Unsloth cliams that the bx121 flags are necessary for optimum performences but their recipe appears to be based on vLLM 0.24 and has specific b121 flags. I tested and the differences are less than 3%, within margin of errors.
I just got unsloth/Qwen3.6-35B-A3B-NVFP4-Fast running from 75 tok/s to 97 tok/s but still missing the --moe-backend flashinfer_b12x and --linear-backend flashinfer_b12x. I also need to change MTP num_speculative_tokens from 3 to 2. Will mess around with it a bit more.
recipe_version: “1”
name: Qwen3.6-35B-A3B-NVFP4-Fast-Unsloth-Marlin-MTP-Solo
description: vLLM serving unsloth/Qwen3.6-35B-A3B-NVFP4-Fast on one DGX Spark node with Marlin MoE backend and MTP speculative decoding (drafter MoE pinned to triton)
model: unsloth/Qwen3.6-35B-A3B-NVFP4-Fast
container: vllm-node
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.6
max_model_len: 262144
max_num_seqs: 4
max_num_batched_tokens: 16384
env:
VLLM_MARLIN_USE_ATOMIC_ADD: 1
CUTE_DSL_ARCH: sm_121a
command: |
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast
–host {host}
–port {port}
–tensor-parallel-size {tensor_parallel}
–trust-remote-code
–kv-cache-dtype fp8
–attention-backend flashinfer
–moe-backend marlin
–speculative-config ‘{“method”:“mtp”,“num_speculative_tokens”:3,“moe_backend”:“triton”}’
–gpu-memory-utilization {gpu_memory_utilization}
–max-model-len {max_model_len}
–max-num-seqs {max_num_seqs}
–max-num-batched-tokens {max_num_batched_tokens}
–enable-chunked-prefill
–async-scheduling
–enable-prefix-caching
–load-format fastsafetensors
–reasoning-parser qwen3
–tool-call-parser qwen3_xml
–enable-auto-tool-choice
Thank you for posting the recipe! I tried this but got an error for speculative-config so changed
*--speculative-config '{{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}}' \\
to*
*--speculative-config '{{"method":"mtp","num_speculative_tokens":3}}' \\*
Now it gets past the error but I get
Download complete: : 67.4kB [00:20, 213kB/s]Error: inference server did not become ready
I run sparkrun benchmark against existing recipes to make sure my setup is OK. See below:
Waiting for inference server on 127.0.0.1:8000…
Note that this could take ~5 minutes!
Waiting for model to finish loading (http://127.0.0.1:8000/v1/models)…
Inference server ready.
Step 2/3: Running benchmark (llama-benchy)…
[11:08:19] Task 1/1 done (10.2s) progress_ui.py:158
0 1 4002.1 52.6 514.8 3
[1/1] d=0 c=1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 100.0% 0:00:00 0:00:10
┏━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ depth ┃ conc ┃ pp t/s ┃ tg t/s ┃ ttfr ms ┃ runs ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ 0 │ 1 │ 4002.1 │ 52.6 │ 514.8 │ 3 │
└───────┴──────┴────────┴────────┴─────────┴──────┘
┏━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ depth ┃ conc ┃ pp t/s ┃ tg t/s ┃ ttfr ms ┃ runs ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ 0 │ 1 │ 4002.1 │ 52.6 │ 514.8 │ 3 │
└───────┴──────┴────────┴────────┴─────────┴──────┘
My first time trying a new recipe on the spark (Asus GX10) so assume I am doing something wrong.
I created a yaml file under ~/.cache/sparkrun/registries/official/official-recipes/qwen3.6/vllm/ and copied the above into that file.
A few recipes that are listed under “sparkrun list” also do not run on my system. How do I debug it?
thanks for any help!