Hello,
One of the biggest factors limiting context length is KV cache memory consumption. NVIDIA Blackwell introduces support for NVFP4 KV Cache, enabling significantly higher token capacity compared to traditional SOTA FP8 kv cache formats.
In this post, I compare NVFP4 and FP8 KV cache using Qwen/Qwen3-4B with SGLang, running on:
- RTX PRO 6000 Blackwell
- DGX Spark
Launch command:
RTX PRO 6000 Blackwell
NVFP4 KV Cache
docker run --rm -it \
--gpus all \
--shm-size 32g \
--ipc host \
--network host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
-e HF_TOKEN="$HF_TOKEN" \
-e HF_HOME=/root/.cache/huggingface \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-v "$HOME/.cache/sglang:/root/.cache/sglang" \
lmsysorg/sglang:dev-cu13 \
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-4B \
--kv-cache-dtype nvfp4 \
--prefill-attention-backend flashinfer \
--decode-attention-backend trtllm_mha \
--disable-radix-cache \
--host 0.0.0.0 \
--port 8005
Server initialization highlights:
Load weight end. elapsed=136.26 s
KV Cache is allocated. dtype: torch.float4_e2m1fn_x2
#tokens: 1,808,192
K size: 36.65 GB
V size: 36.65 GB
max_total_num_tokens=1808192
Benchmark command:
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8005 \
--model Qwen/Qwen3-4B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--random-range-ratio 1.0 \
--num-prompts 100 \
--max-concurrency 100
FP8 KV Cache
docker run --rm -it \
--gpus all \
--shm-size 32g \
--ipc host \
--network host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
-e HF_TOKEN="$HF_TOKEN" \
-e HF_HOME=/root/.cache/huggingface \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-v "$HOME/.cache/sglang:/root/.cache/sglang" \
lmsysorg/sglang:dev-cu13 \
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-4B \
--kv-cache-dtype fp8_e4m3 \
--prefill-attention-backend flashinfer \
--decode-attention-backend trtllm_mha \
--disable-radix-cache \
--host 0.0.0.0 \
--port 8005
Server initialization highlights:
KV Cache is allocated. dtype: torch.float8_e4m3fn
#tokens: 1,067,328
max_total_num_tokens=1067328
The most interesting observation is that NVFP4 increases KV cache capacity by nearly 70%. 1.69x more KV cache capacity
DGX SPARK
NVFP4 KV Cache
KV Cache is allocated. dtype: torch.float4_e2m1fn_x2
#tokens: 2,309,504
max_total_num_tokens=2309504
FP8 KV Cache
KV Cache is allocated. dtype: torch.float8_e4m3fn
#tokens: 1,371,456
max_total_num_tokens=1371456
Again, NVFP4 significantly expands available KV cache capacity. 1.68x more KV cache capacity.
Production deployments should therefore validate model quality and task-specific accuracy before enabling aggressive KV cache quantization. For users looking to maximize context length and request density on Blackwell systems, these results suggest that NVFP4 KV cache offers a highly attractive capacity-to-performance tradeoff compared with FP8.