NVFP4 vs FP8 KV Cache on RTX PRO 6000 Blackwell and DGX Spark

Hello,

One of the biggest factors limiting context length is KV cache memory consumption. NVIDIA Blackwell introduces support for NVFP4 KV Cache, enabling significantly higher token capacity compared to traditional SOTA FP8 kv cache formats.

In this post, I compare NVFP4 and FP8 KV cache using Qwen/Qwen3-4B with SGLang, running on:

  • RTX PRO 6000 Blackwell
  • DGX Spark

Launch command:

RTX PRO 6000 Blackwell

NVFP4 KV Cache

docker run --rm -it \
  --gpus all \
  --shm-size 32g \
  --ipc host \
  --network host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e HF_TOKEN="$HF_TOKEN" \
  -e HF_HOME=/root/.cache/huggingface \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v "$HOME/.cache/sglang:/root/.cache/sglang" \
  lmsysorg/sglang:dev-cu13 \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-4B \
    --kv-cache-dtype nvfp4 \
    --prefill-attention-backend flashinfer \
    --decode-attention-backend trtllm_mha \
    --disable-radix-cache \
    --host 0.0.0.0 \
    --port 8005

Server initialization highlights:

Load weight end. elapsed=136.26 s
KV Cache is allocated. dtype: torch.float4_e2m1fn_x2
#tokens: 1,808,192
K size: 36.65 GB
V size: 36.65 GB

max_total_num_tokens=1808192

Benchmark command:

python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8005 \
  --model Qwen/Qwen3-4B \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --random-range-ratio 1.0 \
  --num-prompts 100 \
  --max-concurrency 100

FP8 KV Cache

docker run --rm -it \
  --gpus all \
  --shm-size 32g \
  --ipc host \
  --network host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e HF_TOKEN="$HF_TOKEN" \
  -e HF_HOME=/root/.cache/huggingface \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v "$HOME/.cache/sglang:/root/.cache/sglang" \
  lmsysorg/sglang:dev-cu13 \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-4B \
    --kv-cache-dtype fp8_e4m3 \
    --prefill-attention-backend flashinfer \
    --decode-attention-backend trtllm_mha \
    --disable-radix-cache \
    --host 0.0.0.0 \
    --port 8005

Server initialization highlights:

KV Cache is allocated. dtype: torch.float8_e4m3fn
#tokens: 1,067,328

max_total_num_tokens=1067328

The most interesting observation is that NVFP4 increases KV cache capacity by nearly 70%. 1.69x more KV cache capacity

DGX SPARK

NVFP4 KV Cache

KV Cache is allocated. dtype: torch.float4_e2m1fn_x2
#tokens: 2,309,504

max_total_num_tokens=2309504

FP8 KV Cache

KV Cache is allocated. dtype: torch.float8_e4m3fn
#tokens: 1,371,456

max_total_num_tokens=1371456

Again, NVFP4 significantly expands available KV cache capacity. 1.68x more KV cache capacity.

Production deployments should therefore validate model quality and task-specific accuracy before enabling aggressive KV cache quantization. For users looking to maximize context length and request density on Blackwell systems, these results suggest that NVFP4 KV cache offers a highly attractive capacity-to-performance tradeoff compared with FP8.

6000 Pro

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     100
Benchmark duration (s):                  18.96
Total input tokens:                      100000
Total input text tokens:                 100000
Total generated tokens:                  100000
Total generated tokens (retokenized):    81963
Request throughput (req/s):              5.27
Input token throughput (tok/s):          5274.96
Output token throughput (tok/s):         5274.96
Peak output token throughput (tok/s):    7300.00
Peak concurrent requests:                100
Total token throughput (tok/s):          10549.92
Concurrency:                             99.92
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   18942.87
Median E2E Latency (ms):                 18942.96
P90 E2E Latency (ms):                    18944.62
P95 E2E Latency (ms):                    18944.91
P99 E2E Latency (ms):                    18945.24
---------------Time to First Token----------------
Mean TTFT (ms):                          1914.97
Median TTFT (ms):                        2035.93
P90 TTFT (ms):                           3421.24
P95 TTFT (ms):                           3421.55
P99 TTFT (ms):                           3496.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.04
Median TPOT (ms):                        16.92
P90 TPOT (ms):                           18.32
P95 TPOT (ms):                           18.58
P99 TPOT (ms):                           18.58
---------------Inter-Token Latency----------------
Mean ITL (ms):                           17.04
Median ITL (ms):                         15.36
P90 ITL (ms):                            17.07
P95 ITL (ms):                            17.23
P99 ITL (ms):                            17.96
Max ITL (ms):                            3377.91
==================================================

DGX Spark

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     100
Benchmark duration (s):                  131.94
Total input tokens:                      100000
Total input text tokens:                 100000
Total generated tokens:                  100000
Total generated tokens (retokenized):    80187
Request throughput (req/s):              0.76
Input token throughput (tok/s):          757.92
Output token throughput (tok/s):         757.92
Peak output token throughput (tok/s):    1300.00
Peak concurrent requests:                100
Total token throughput (tok/s):          1515.84
Concurrency:                             99.91
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   131815.09
Median E2E Latency (ms):                 131815.45
P90 E2E Latency (ms):                    131821.44
P95 E2E Latency (ms):                    131821.83
P99 E2E Latency (ms):                    131822.42
---------------Time to First Token----------------
Mean TTFT (ms):                          13279.95
Median TTFT (ms):                        13331.13
P90 TTFT (ms):                           22202.60
P95 TTFT (ms):                           24038.19
P99 TTFT (ms):                           24501.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          118.65
Median TPOT (ms):                        118.60
P90 TPOT (ms):                           127.52
P95 TPOT (ms):                           129.18
P99 TPOT (ms):                           131.04
---------------Inter-Token Latency----------------
Mean ITL (ms):                           118.65
Median ITL (ms):                         115.80
P90 ITL (ms):                            128.45
P95 ITL (ms):                            129.89
P99 ITL (ms):                            131.25
Max ITL (ms):                            23957.08
==================================================