MXFP6 W6A8 on RTX 5090 / SM120: Qwen3.8-27B quality, throughput, and memory trade-offs

We have been working on a native MXFP6 W6A8 GEMM path for SM120 and wanted to measure where it sits between FP8 and NVFP4 in a complete serving stack.

This is not a claim that MXFP6 is always the best format. FP8 remains closest to BF16; NVFP4 remains fastest and smallest. The question is whether a 6-bit path provides a useful middle operating point.

Format and kernel path

  • MXFP6 weights: packed OCP E3M2, with one UE8M0 scale per 32 values
  • BF16 or FP16 activations dynamically quantized to E4M3
  • Native SM120 W6A8 GEMM with FP32 accumulation
  • NVFP4 baseline: standard W4A4, with groups of 16

The comparison uses complete services, so it includes activation quantization, scheduling, communication, and attention rather than extrapolating from GEMM microbenchmarks.

Experimental contract

  • Qwen3.8-27B on the same PIX-connected pair of RTX 5090 GPUs, TP2
  • Frozen vLLM 0.25.1 integration image and the same 90% memory budget
  • Public BF16 and official FP8 checkpoints
  • MXFP6 and NVFP4 quantize the same 400 matrices; lm_head remains BF16
  • Fixed serving sweep: ISL 1024, OSL 256, unlimited request rate, concurrency 1/4/8/16/32
  • Two fresh service lifecycles per format, with reversed format and concurrency order
  • Fidelity set: 256 teacher-forced records, 64 each from GSM8K, HumanEval, HellaSwag, and CMMLU

BF16 token-logprob fidelity

Format Sample-mean MAE vs BF16 95% CI Top-1 token agreement
FP8 0.05719 [0.05042, 0.06431] 97.47%
MXFP6 0.09078 [0.08250, 0.09932] 96.57%
NVFP4 0.17661 [0.15886, 0.19576] 93.27%

MXFP6’s sample-mean MAE is 48.6% lower than NVFP4’s. FP8 remains closest to BF16. These are fidelity metrics, not downstream task accuracy or human preference.

TP2 serving throughput

Max concurrency FP8 output tok/s MXFP6 output tok/s NVFP4 output tok/s MXFP6 vs FP8
1 84.98 100.49 120.70 +18.26%
4 286.81 345.32 411.70 +20.40%
8 494.06 598.47 694.44 +21.13%
16 767.30 930.59 1,089.38 +21.28%
32 1,097.49 1,343.67 1,554.27 +22.43%

MXFP6 is 18.3%-22.4% faster than FP8 and 13.5%-16.7% slower than NVFP4 across the sweep. All 30 measured points completed every request, produced the requested token counts, and logged the expected quantized kernel path. The largest two-run throughput range was 1.35% of its mean.

Serving memory

Format Model load GiB/GPU Available KV GiB/GPU GPU KV-cache tokens
FP8 14.09 12.46 236,658
MXFP6 11.46 14.99 284,899
NVFP4 9.05 17.47 332,003

Relative to the measured FP8-to-NVFP4 endpoints, MXFP6 occupies 28.1% of the BF16-fidelity gap while recovering 43.4%-53.9% of the throughput gain, 52.2% of the model-load saving, and 50.6% of the added KV capacity. This endpoint normalization is not a composite score.

The practical reading is:

  • If NVFP4 passes a service’s quality gate, it provides more throughput and capacity.
  • If the closest measured approximation to BF16 is required, FP8 remains the better fit when it fits.
  • MXFP6 is relevant when FP8 leaves performance or capacity on the table, but NVFP4 moves too far from BF16 for that deployment.

Full report, exact tables, limitations, and machine-readable artifact links:

The serving snapshot is specific to Qwen3.8-27B, RTX 5090, the recorded TP sizes, and the frozen runtime. The internal image digest makes it auditable, but this exact integration image is not a public reproducer.

Feedback on the SM120 kernel path, the measurement contract, or useful workloads for an independent reproduction would be welcome.

Very interesting work!

Did you look at E2M3 as well?

As the significant savings come from memory bandwidth/capacity reductions rather than increased MMA throughput, did you look at how MXFP4/FP8 W4A8 compares?

Does the NVFP4 quantisation used always map abs(amax(block16)) to 6.0 in fp4, or is it more sophisticated than that?

Thanks — these are good questions.

I did not test E2M3 in this sweep. The current MXFP6 implementation and checkpoint path are E3M2-specific, using UE8M0 scales over groups of 32. E2M3 keeps the same 6-bit payload and scale-vector size, so it would be a useful quality ablation, but it requires a separate quantizer/kernel/checkpoint path. I would rather measure it than infer the result from E3M2.

I also did not test MXFP4 weights with FP8 activations (W4A8). The practical reason is that the NVFP4 W4A4 path reported here was already outside our deployment fidelity requirement, so we did not prioritize additional 4-bit weight variants such as MXFP4/FP8 in this deployment-driven sweep.

That is an engineering prioritization rather than evidence that MXFP4/FP8 would have the same fidelity as NVFP4 W4A4. I agree that W4A8 would still be a useful experimental control for separating weight bandwidth/capacity savings from the W4A4 MMA path; it was simply outside the scope we prioritized once FP4 failed our quality gate.

For the NVFP4 checkpoint used in the reported measurements, the final conversion used the standard compressed-tensors NVFP4 oneshot path with 20 calibration samples (seed 42, maximum sequence length 2048).

The short answer to the block-scaling question is: max-based, but not a literal invariant that the effective normalized block maximum is always exactly 6.

Weights use E2M1 with group size 16, an FP8 E4M3 local scale and an FP32 global scale. In the implementation used here, the weight global scale is derived as

g = 448 * 6 / tensor_absmax

and the local block scale is approximately

s_block = round_E4M3(g * block_absmax / 6).

The effective scale is s_block / g. Therefore, before E4M3 scale rounding, the block absmax targets the E2M1 endpoint of 6; after scale quantization, it is not guaranteed to remain exactly 6. No GPTQ, MSE/ScaleSweep or other per-block scale search was used.

For activations, the global scale was calibrated from those samples, while the block-16 local scales are computed dynamically at runtime.