We have been working on a native MXFP6 W6A8 GEMM path for SM120 and wanted to measure where it sits between FP8 and NVFP4 in a complete serving stack.
This is not a claim that MXFP6 is always the best format. FP8 remains closest to BF16; NVFP4 remains fastest and smallest. The question is whether a 6-bit path provides a useful middle operating point.
Format and kernel path
- MXFP6 weights: packed OCP E3M2, with one UE8M0 scale per 32 values
- BF16 or FP16 activations dynamically quantized to E4M3
- Native SM120 W6A8 GEMM with FP32 accumulation
- NVFP4 baseline: standard W4A4, with groups of 16
The comparison uses complete services, so it includes activation quantization, scheduling, communication, and attention rather than extrapolating from GEMM microbenchmarks.
Experimental contract
- Qwen3.8-27B on the same PIX-connected pair of RTX 5090 GPUs, TP2
- Frozen vLLM 0.25.1 integration image and the same 90% memory budget
- Public BF16 and official FP8 checkpoints
- MXFP6 and NVFP4 quantize the same 400 matrices; lm_head remains BF16
- Fixed serving sweep: ISL 1024, OSL 256, unlimited request rate, concurrency 1/4/8/16/32
- Two fresh service lifecycles per format, with reversed format and concurrency order
- Fidelity set: 256 teacher-forced records, 64 each from GSM8K, HumanEval, HellaSwag, and CMMLU
BF16 token-logprob fidelity
| Format | Sample-mean MAE vs BF16 | 95% CI | Top-1 token agreement |
|---|---|---|---|
| FP8 | 0.05719 | [0.05042, 0.06431] | 97.47% |
| MXFP6 | 0.09078 | [0.08250, 0.09932] | 96.57% |
| NVFP4 | 0.17661 | [0.15886, 0.19576] | 93.27% |
MXFP6’s sample-mean MAE is 48.6% lower than NVFP4’s. FP8 remains closest to BF16. These are fidelity metrics, not downstream task accuracy or human preference.
TP2 serving throughput
| Max concurrency | FP8 output tok/s | MXFP6 output tok/s | NVFP4 output tok/s | MXFP6 vs FP8 |
|---|---|---|---|---|
| 1 | 84.98 | 100.49 | 120.70 | +18.26% |
| 4 | 286.81 | 345.32 | 411.70 | +20.40% |
| 8 | 494.06 | 598.47 | 694.44 | +21.13% |
| 16 | 767.30 | 930.59 | 1,089.38 | +21.28% |
| 32 | 1,097.49 | 1,343.67 | 1,554.27 | +22.43% |
MXFP6 is 18.3%-22.4% faster than FP8 and 13.5%-16.7% slower than NVFP4 across the sweep. All 30 measured points completed every request, produced the requested token counts, and logged the expected quantized kernel path. The largest two-run throughput range was 1.35% of its mean.
Serving memory
| Format | Model load GiB/GPU | Available KV GiB/GPU | GPU KV-cache tokens |
|---|---|---|---|
| FP8 | 14.09 | 12.46 | 236,658 |
| MXFP6 | 11.46 | 14.99 | 284,899 |
| NVFP4 | 9.05 | 17.47 | 332,003 |
Relative to the measured FP8-to-NVFP4 endpoints, MXFP6 occupies 28.1% of the BF16-fidelity gap while recovering 43.4%-53.9% of the throughput gain, 52.2% of the model-load saving, and 50.6% of the added KV capacity. This endpoint normalization is not a composite score.
The practical reading is:
- If NVFP4 passes a service’s quality gate, it provides more throughput and capacity.
- If the closest measured approximation to BF16 is required, FP8 remains the better fit when it fits.
- MXFP6 is relevant when FP8 leaves performance or capacity on the table, but NVFP4 moves too far from BF16 for that deployment.
Full report, exact tables, limitations, and machine-readable artifact links:
The serving snapshot is specific to Qwen3.8-27B, RTX 5090, the recorded TP sizes, and the frozen runtime. The internal image digest makes it auditable, but this exact integration image is not a public reproducer.
Feedback on the SM120 kernel path, the measurement contract, or useful workloads for an independent reproduction would be welcome.
