Benchmark Report: unsloth/Qwen3.6-35B-A3B-NVFP4-Fast vs nvidia/Qwen3.6-35B-A3B-NVFP4

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

Hi everyone,

I ran a comparison of different quantized versions of Qwen3.6-35B-A3B-NVFP4 on vLLM 0.25.0:

Starting the vllm server with flashinfer_b12x backend

For the Unsloth model:

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast --moe-backend flashinfer_b12x

Benchmark command:

vllm bench serve \
  --model unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --request-rate 100 \
  --num-prompts 100 \
  --ignore-eos

Results: unsloth/Qwen3.6-35B-A3B-NVFP4-Fast

Single RTX 6000 Pro

============ Serving Benchmark Result ============
Successful requests:                     100
Failed requests:                         0
Request rate configured (RPS):           100.00
Benchmark duration (s):                  31.07
Total input tokens:                      100000
Total generated tokens:                  100000
Request throughput (req/s):              3.22
Output token throughput (tok/s):         3218.35
Peak output token throughput (tok/s):    3800.00
Peak concurrent requests:                100.00
Total token throughput (tok/s):          6436.70
---------------Time to First Token----------------
Mean TTFT (ms):                          1141.97
Median TTFT (ms):                        1107.41
P99 TTFT (ms):                           1814.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.31
Median TPOT (ms):                        29.33
P99 TPOT (ms):                           30.29
---------------Inter-token Latency----------------
Mean ITL (ms):                           29.31
Median ITL (ms):                         28.69
P99 ITL (ms):                            32.08
==================================================

DGX Spark

============ Serving Benchmark Result ============
Successful requests:                     100
Failed requests:                         0
Request rate configured (RPS):           100.00
Benchmark duration (s):                  229.44
Total input tokens:                      100000
Total generated tokens:                  100000
Request throughput (req/s):              0.44
Output token throughput (tok/s):         435.84
Peak output token throughput (tok/s):    600.00
Peak concurrent requests:                100.00
Total token throughput (tok/s):          871.69
---------------Time to First Token----------------
Mean TTFT (ms):                          13573.71
Median TTFT (ms):                        13400.97
P99 TTFT (ms):                           24417.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          212.83
Median TPOT (ms):                        213.31
P99 TPOT (ms):                           219.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           212.83
Median ITL (ms):                         210.97
P99 ITL (ms):                            470.14
==================================================

Results: nvidia/Qwen3.6-35B-A3B-NVFP4

Then test nvidia/Qwen3.6-35B-A3B-NVFP4 · Hugging Face with default Marlin backend:

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
    --port 8000 \
    --quantization modelopt \
    --max-model-len 262144 \
    --reasoning-parser qwen3

Single 6000 Pro:

============ Serving Benchmark Result ============
Successful requests:                     100
Failed requests:                         0
Request rate configured (RPS):           100.00
Benchmark duration (s):                  45.73
Total input tokens:                      100000
Total generated tokens:                  100000
Request throughput (req/s):              2.19
Output token throughput (tok/s):         2186.98
Peak output token throughput (tok/s):    4100.00
Peak concurrent requests:                100.00
Total token throughput (tok/s):          4373.96
---------------Time to First Token----------------
Mean TTFT (ms):                          18277.23
Median TTFT (ms):                        18218.49
P99 TTFT (ms):                           19379.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.83
Median TPOT (ms):                        26.88
P99 TPOT (ms):                           28.10
---------------Inter-token Latency----------------
Mean ITL (ms):                           26.83
Median ITL (ms):                         25.66
P99 ITL (ms):                            30.04
==================================================

DGX Spark

============ Serving Benchmark Result ============
Successful requests:                     100
Failed requests:                         0
Request rate configured (RPS):           100.00
Benchmark duration (s):                  247.38
Total input tokens:                      100000
Total generated tokens:                  100000
Request throughput (req/s):              0.40
Output token throughput (tok/s):         404.24
Peak output token throughput (tok/s):    600.00
Peak concurrent requests:                100.00
Total token throughput (tok/s):          808.48
---------------Time to First Token----------------
Mean TTFT (ms):                          38299.51
Median TTFT (ms):                        37504.90
P99 TTFT (ms):                           54582.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          206.14
Median TPOT (ms):                        207.26
P99 TPOT (ms):                           216.33
---------------Inter-token Latency----------------
Mean ITL (ms):                           206.14
Median ITL (ms):                         196.58
P99 ITL (ms):                            717.73
==================================================

On both platforms, Unsloth’s NVFP4-Fast variant significantly outperformed the NVIDIA release in TTFT and overall throughput.

2

Hello,

How did you manage to make flashinfer_b12x work on DGX Spark ?

Thank you

3

from there blog for Qwen3.6:

Qwen3.6 - How to Run Locally | Unsloth Documentation

DGX Spark with NVFP4 quants

To ensure DGX Spark has the correct kernels (or you will get 2x SLOWER inference), first check:

Copy

python -c "
import torch; from vllm.utils.flashinfer import has_flashinfer_b12x_gemm as g, has_flashinfer_b12x_moe as m
cap = torch.cuda.get_device_capability(); print('cap', cap, '| b12x gemm', g(), '| b12x moe', m()); assert cap[0] == 12 and g() and m(), 'b12x unavailable: serving would degrade to marlin W4A16'"

which should NOT error out - if it did, please update vllm or reinstall via:

Copy

uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto

Then to serve in vLLM for DGX Spark:

Copy

export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast --moe-backend flashinfer_b12x

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

Copy

sudo apt-get update
sudo apt-get install -y ffmpeg
4

Ah without specific linear backend and without mtp you got better performances ?

Ive tested without linear backend and with mtp with triton but I got bad performances.

Ill try that thank you 🙂

5

I can’t run vllm 0.25.x at all - both in python env or docker, it’s always stuck on Waiting for 1 local, 0 remote core engine proc(s) to start. Please help.

6

nvidia/Qwen3.6-35B-A3B-NVFP4 on Nvidia B200

============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Benchmark duration (s):                  96.62
Total input tokens:                      1024000
Total generated tokens:                  128000
Request throughput (req/s):              10.35
Output token throughput (tok/s):         1324.82
Peak output token throughput (tok/s):    12675.00
Peak concurrent requests:                1000.00
Total token throughput (tok/s):          11923.42
---------------Time to First Token----------------
Mean TTFT (ms):                          13141.46
Median TTFT (ms):                        12746.76
P99 TTFT (ms):                           26193.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          482.87
Median TPOT (ms):                        571.18
P99 TPOT (ms):                           649.85
---------------Inter-token Latency----------------
Mean ITL (ms):                           483.25
Median ITL (ms):                         194.56
P99 ITL (ms):                            1595.73
==================================================