Qwen3.8-27B on DGX Spark using vllm: NVFP4 vs FP8 performance

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hello, I benchmarked two quantizations of the same Qwen3.8-27B on a single Nvidia DGX Spark

  • Qwen/Qwen3.8-27B-FP8 — fine-grained FP8 (block 128, e4m3), official Qwen FP8 build.
  • unsloth/Qwen3.8-27B-NVFP4 — Unsloth Dynamic NVFP4 (4-bit MLP + 8-bit attention + FP8 KV cache).

The results show a consistent 30–34 % faster generation throughput for the NVFP4 model from Unsloth,
plus a smaller model and lower KV-cache memory, especially for decode-heavy workloads.

Both runs use vLLM 0.27.1 with 16 concurrent prompts and identical flags,

Workloads

  • Prompt-heavy: 8000 input / 1000 output tokens
  • Decode-heavy: 1000 input / 8000 output tokens
  • Balanced: 1000 input / 1000 output tokens

Command used for each run (model and lengths change per scenario):

vllm serve <model> --port 8000 --dtype auto

vllm bench serve \
  --model <model> \
  --dataset-name random \
  --random-input-len <IN> \
  --random-output-len <OUT> \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos

Results summary

Output token throughput (16 concurrent requests)

Scenario FP8 (tok/s) NVFP4 (tok/s) NVFP4 gain
Prompt-heavy (8k→1k) 65.58 87.91 (peak 128) +34 %
Decode-heavy (1k→8k) 99.47 132.07 (peak 144) +33 %
Balanced (1k→1k) 104.44 134.41 (peak 144) +29 %

Latency & efficiency

Metric FP8 NVFP4
Prompt-heavy TTFT (mean) 36 904 ms 29 815 ms
Decode-heavy TPOT (mean) 160.19 ms 120.60 ms
Balanced TPOT (mean) 147.90 ms 114.73 ms
Decode-heavy KV-cache usage ~15 % ~5.6 %
Model size 30.9 GB 23.4 GB

FP8 - Qwen/Qwen3.8-27B-FP8

Prompt-heavy

vllm bench serve --model Qwen/Qwen3.8-27B-FP8 \
  --dataset-name random \
  --random-input-len 8000 --random-output-len 1000 \
  --request-rate 10000 --num-prompts 16 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Request rate configured (RPS):           10000.00
Benchmark duration (s):                  243.97
Total input tokens:                      128000
Total generated tokens:                  16000
Request throughput (req/s):              0.07
Output token throughput (tok/s):         65.58
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          590.25
---------------Time to First Token----------------
Mean TTFT (ms):                          36904.45
Median TTFT (ms):                        36629.98
P99 TTFT (ms):                           70322.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          203.19
Median TPOT (ms):                        203.67
P99 TPOT (ms):                           230.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           203.19
Median ITL (ms):                         174.58
P99 ITL (ms):                            1181.35
==================================================

Decode-heavy

vllm bench serve --model Qwen/Qwen3.8-27B-FP8 \
  --dataset-name random \
  --random-input-len 1000 --random-output-len 8000 \
  --request-rate 10000 --num-prompts 16 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Request rate configured (RPS):           10000.00
Benchmark duration (s):                  1286.83
Total input tokens:                      16000
Total generated tokens:                  128000
Request throughput (req/s):              0.01
Output token throughput (tok/s):         99.47
Peak output token throughput (tok/s):    112.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          111.90
---------------Time to First Token----------------
Mean TTFT (ms):                          4929.30
Median TTFT (ms):                        5038.53
P99 TTFT (ms):                           7935.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          160.19
Median TPOT (ms):                        160.18
P99 TPOT (ms):                           160.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           160.19
Median ITL (ms):                         159.83
P99 ITL (ms):                            176.42
==================================================

Balanced

vllm bench serve --model Qwen/Qwen3.8-27B-FP8 \
  --dataset-name random \
  --random-input-len 1000 --random-output-len 1000 \
  --request-rate 10000 --num-prompts 16 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Request rate configured (RPS):           10000.00
Benchmark duration (s):                  153.20
Total input tokens:                      16000
Total generated tokens:                  16000
Request throughput (req/s):              0.10
Output token throughput (tok/s):         104.44
Peak output token throughput (tok/s):    112.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          208.88
---------------Time to First Token----------------
Mean TTFT (ms):                          4930.32
Median TTFT (ms):                        5044.74
P99 TTFT (ms):                           7925.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          147.90
Median TPOT (ms):                        147.80
P99 TPOT (ms):                           151.19
---------------Inter-token Latency----------------
Mean ITL (ms):                           147.90
Median ITL (ms):                         145.57
P99 ITL (ms):                            148.39
==================================================

NVFP4 - unsloth/Qwen3.8-27B-NVFP4

Prompt-heavy

vllm bench serve --model unsloth/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 8000 --random-output-len 1000 \
  --request-rate 10000 --num-prompts 16 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Request rate configured (RPS):           10000.00
Benchmark duration (s):                  182.01
Total input tokens:                      128000
Total generated tokens:                  16000
Request throughput (req/s):              0.09
Output token throughput (tok/s):         87.91
Peak output token throughput (tok/s):    128.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          791.16
---------------Time to First Token----------------
Mean TTFT (ms):                          29814.55
Median TTFT (ms):                        30020.07
P99 TTFT (ms):                           54669.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          149.37
Median TPOT (ms):                        149.31
P99 TPOT (ms):                           171.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           149.37
Median ITL (ms):                         128.22
P99 ITL (ms):                            893.95
==================================================

Decode-heavy

vllm bench serve --model unsloth/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1000 --random-output-len 8000 \
  --request-rate 10000 --num-prompts 16 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Request rate configured (RPS):           10000.00
Benchmark duration (s):                  969.15
Total input tokens:                      16000
Total generated tokens:                  128000
Request throughput (req/s):              0.02
Output token throughput (tok/s):         132.07
Peak output token throughput (tok/s):    144.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          148.58
---------------Time to First Token----------------
Mean TTFT (ms):                          4047.64
Median TTFT (ms):                        4138.09
P99 TTFT (ms):                           6493.16
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          120.60
Median TPOT (ms):                        120.59
P99 TPOT (ms):                           120.94
---------------Inter-token Latency----------------
Mean ITL (ms):                           120.60
Median ITL (ms):                         120.36
P99 ITL (ms):                            129.30
==================================================

Balanced

vllm bench serve --model unsloth/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1000 --random-output-len 1000 \
  --request-rate 10000 --num-prompts 16 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Request rate configured (RPS):           10000.00
Benchmark duration (s):                  119.04
Total input tokens:                      16000
Total generated tokens:                  16000
Request throughput (req/s):              0.13
Output token throughput (tok/s):         134.41
Peak output token throughput (tok/s):    144.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          268.82
---------------Time to First Token----------------
Mean TTFT (ms):                          4046.00
Median TTFT (ms):                        4133.61
P99 TTFT (ms):                           6502.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          114.73
Median TPOT (ms):                        114.65
P99 TPOT (ms):                           117.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           114.73
Median ITL (ms):                         112.64
P99 ITL (ms):                            115.53
==================================================
2

I benchmarked the two quants of Qwen3.8-27B on an NVIDIA Jetson Thor developer kit:

Latency comparison tables

Time to First Token (TTFT, ms) — lower is better

Scenario FP8 NVFP4 reduction
Prompt-heavy 43170 27383 −37%
Decode-heavy 9358 3801 −59%
Balanced 9497 3794 −60%

Time per Output Token (TPOT, ms) — lower is better

Scenario FP8 NVFP4 reduction
Prompt-heavy 199.75 139.41 −30%
Decode-heavy 153.85 112.99 −27%
Balanced 144.58 107.10 −26%

Total token throughput (tok/s) — higher is better

Scenario FP8 NVFP4 Δ
Prompt-heavy 587.02 854.35 +45.5%
Decode-heavy 116.77 159.52 +36.6%
Balanced 212.79 295.65 +39.0%

NVFP4 wins on every metric* on Jetson Thor — higher throughput and lower latency.

3

Hello @shahizat!

Based on the title and content of your topic, it looks like it may receive better visibility and feedback in a different category. We took the liberty of moving it for you.

If this was an incorrect assessment, please send me a direct message.

Disclaimer: this moderation suggestion and message were generated with AI assistance.

4

Hi @MarkusHoHo, It is more about the DGX Spark, but I tested it on the jetson thor as well.

5

can you test with c=1 instead of 16? thanks~

8

Honestly, I am not sure why the bot moved this to Jetson in the first place. Since it is more relevant to GB10, I move it back. I hope that is ok.

9

could you give us the recipe for eugr vllm for starting both models you use.
thanks in advance

10

Did you do a run with just 1 vs 16 parallel prompts?

I am only getting 10-12 tokens for a single prompt with medium thinking set in real world coding tasks.

Below is my recipe for XQDev since he asked:

docker run -it --gpus all --ipc=host -p 8000:8000 \
-e HF_TOKEN=“${HF_TOKEN}” \
-e MAX_JOBS=4 \
-e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=“0” \
-e VLLM_USE_DEEP_GEMM=“1” \
-e CUTE_DSL_ARCH=sm_121a \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest Qwen/Qwen3.8-27B-FP8 \
–host 0.0.0.0 \
–port 8000 \
–tensor-parallel-size 1 \
–kv-cache-dtype fp8 \
–safetensors-load-strategy lazy \
–enable-auto-tool-choice \
–tool-call-parser qwen3_coder \
–reasoning-parser qwen3 \
–served-model-name qwen \
–speculative-config ‘{“method”:“mtp”,“num_speculative_tokens”:2}’ \
–max-num-seqs 4 \
–max-model-len 212992 \
–max-num-batched-tokens 8192 \
–gpu-memory-utilization 0.88 \
–enable-chunked-prefill \
–trust-remote-code
12

Nice thread! I got similar results comparing (+25% on NVFP4) if anyone wants to see a side by side: https://youtu.be/XwKOdegg81A. I also tested this clustered to see performance and it wasn’t pretty: https://www.youtube.com/watch?v=65TTAbzGIWI&list=PLVZuUjW8N01s&index=6