By enabling calibration-assisted quantization, you can consistently achieve high-quality results. The purpose of this MXFP4 online quantization patch is to accelerate inference processing when only BF16 format checkpoints are available. By performing the BF16→FP4 conversion on-the-fly during model loading, you can improve processing throughput.
If a suitable pre-quantized checkpoint with proper calibration already exists for the target model (e.g., in FP8 format), it is recommended to use that instead. This will not only yield higher-quality results but also improve model loading speed.
You’re suggesting loading a BF16 model in a 4-node environment, quantizing it to MXFP4, then saving the quantized weights to load them in a 2-node environment. While this might be challenging in practice, it could certainly make for an interesting research project.
High performance is expected from NVFP4, but MXFP4 consistently delivers the best results. How can we maintain quality without sacrificing it? This is just an idea, perhaps not the best one. Thank you for your support.
The most frequent cause of output artifacts in this patch’s MXFP4 quantization occurs when attention layer quantization is not performed correctly. For models like gpt-oss-120b, you must properly configure the modules_to_not_convert list in the quantization configuration file to preserve attention weights in BF16 format. Quantizing these weights to FP4 significantly degrades output quality. Any configuration inconsistencies in this situation will lead to
behavior that differs from my expectations.
There were issues with this patch regarding the from_config() modification in mxfp4.py (specifically the approximately three lines of code that handle loading modules_to_not_convert):
What I found:
When this part of the patch is applied, decode speed drops significantly — on TP=2, from 80 tok/s down to 52 tok/s.
When this part is NOT applied, full speed is maintained, but there is a possibility of quality degradation. (This may be related to an issue that has already been reported.)
What I’m doing:
I’m investigating the quality degradation issue further.
Recommendation:
Please hold off on applying the patch for now until I resolve this. I will update this thread once a fix is ready.
I apologize for the repeated changes — I’m not very experienced with publishing patches publicly, and circumstances have made it difficult to keep up. I’m sorry for the inconvenience to those who have tried multiple iterations.
I struggled with precision issues, but after exhaustively testing all reasonable precision combinations for each layer, I was able to find the combination that maximizes performance while maintaining output quality.
This combination produces correct responses for all short and long prompts I prepared, and achieves the highest throughput.
The previous version ran o_proj in MXFP4 as well, but it was found to potentially degrade quality, so it now runs in FP8.
Layer
Precision
Kernel
bytes/param
Notes
MoE experts (w1, w2, w3)
MXFP4 (E2M1)
Marlin FP4
0.5 + scale
Pre-quantized or online quantized
QKV (q_proj, k_proj, v_proj)
MXFP4 (E2M1)
Marlin FP4
0.5 + scale
Softmax normalizes quantization error
o_proj
FP8 (E4M3)
Marlin FP8
1.0
E2M1 causes repetition loops in long generation
lm_head
MXFP4 (E2M1)
Marlin FP4
0.5 + scale
Falls back to BF16 when tie_word_embeddings=True
embed_tokens
BF16
–
2.0
Embedding gather, not a GEMM
router
BF16
–
2.0
Negligible size (~13 MB)
layer_norm
BF16
–
2.0
Negligible size (~0.4 MB)
The following throughput is achieved with a single request (num-prompts=1).
openai/gpt-oss-120b
Configuration
TPOT (ms)
Throughput (tok/s)
TTFT (ms)
Vanilla MXFP4 TP=1
–
– (broken on SM121)
–
Vanilla MXFP4 TP=2
19.09
48.66
206.48
Patched TP=1
15.87
61.78
56.39
Patched TP=2
12.33
79.28
48.85
Qwen3.5-35B-A3B
Configuration
TPOT (ms)
Throughput (tok/s)
TTFT (ms)
Vanilla BF16 TP=1
32.93
28.41
327.34
Vanilla BF16 TP=2
21.68
39.71
469.77
Patched MXFP4 TP=1
15.18
60.06
203.52
Patched MXFP4 TP=2
12.92
70.83
166.28
Here are the llama-benchy results for openai/gpt-oss-120b.
model
test
t/s
peak t/s
ttfr (ms)
est_ppt (ms)
e2e_ttft (ms)
openai/gpt-oss-120b
pp2048
6034.06 ± 44.64
340.56 ± 2.52
339.43 ± 2.52
380.37 ± 2.52
openai/gpt-oss-120b
tg32
78.20 ± 0.50
81.00 ± 0.52
openai/gpt-oss-120b
ctx_pp @ d4096
6573.50 ± 24.97
624.25 ± 2.36
623.12 ± 2.36
661.66 ± 2.65
openai/gpt-oss-120b
ctx_tg @ d4096
76.02 ± 2.70
78.74 ± 2.80
openai/gpt-oss-120b
pp2048 @ d4096
5018.79 ± 6.84
409.20 ± 0.56
408.07 ± 0.56
446.24 ± 0.67
openai/gpt-oss-120b
tg32 @ d4096
76.90 ± 0.43
79.66 ± 0.44
openai/gpt-oss-120b
ctx_pp @ d8192
6261.30 ± 4.46
1309.49 ± 0.93
1308.36 ± 0.93
1344.05 ± 1.58
openai/gpt-oss-120b
ctx_tg @ d8192
75.84 ± 0.12
78.56 ± 0.12
openai/gpt-oss-120b
pp2048 @ d8192
4205.96 ± 20.02
488.07 ± 2.31
486.94 ± 2.31
522.31 ± 2.22
openai/gpt-oss-120b
tg32 @ d8192
75.48 ± 0.68
78.19 ± 0.71
openai/gpt-oss-120b
ctx_pp @ d16384
5415.59 ± 3.03
3026.48 ± 1.69
3025.34 ± 1.69
3059.52 ± 1.69
openai/gpt-oss-120b
ctx_tg @ d16384
72.66 ± 0.23
75.26 ± 0.24
openai/gpt-oss-120b
pp2048 @ d16384
3327.71 ± 29.32
616.62 ± 5.46
615.49 ± 5.46
648.86 ± 5.55
openai/gpt-oss-120b
tg32 @ d16384
70.98 ± 2.09
73.52 ± 2.17
openai/gpt-oss-120b
ctx_pp @ d32768
4120.12 ± 10.69
7954.36 ± 20.63
7953.22 ± 20.63
7986.08 ± 21.64
openai/gpt-oss-120b
ctx_tg @ d32768
68.01 ± 0.75
70.44 ± 0.77
openai/gpt-oss-120b
pp2048 @ d32768
2327.84 ± 5.05
880.92 ± 1.91
879.79 ± 1.91
910.20 ± 1.95
openai/gpt-oss-120b
tg32 @ d32768
67.66 ± 0.63
70.08 ± 0.65
openai/gpt-oss-120b
ctx_pp @ d65535
2766.30 ± 9.41
23691.93 ± 80.45
23690.79 ± 80.45
23714.84 ± 78.42
openai/gpt-oss-120b
ctx_tg @ d65535
56.91 ± 6.43
59.11 ± 6.44
openai/gpt-oss-120b
pp2048 @ d65535
1462.52 ± 1.95
1401.46 ± 1.86
1400.33 ± 1.86
1422.08 ± 0.55
openai/gpt-oss-120b
tg32 @ d65535
60.81 ± 0.17
62.98 ± 0.18
openai/gpt-oss-120b
ctx_pp @ d100000
2059.25 ± 10.66
48563.70 ± 252.37
48562.56 ± 252.37
48585.46 ± 254.11
openai/gpt-oss-120b
ctx_tg @ d100000
57.42 ± 1.18
59.47 ± 1.23
openai/gpt-oss-120b
pp2048 @ d100000
1031.87 ± 2.10
1985.88 ± 4.03
1984.75 ± 4.03
2007.05 ± 3.43
openai/gpt-oss-120b
tg32 @ d100000
56.40 ± 1.34
58.42 ± 1.38
llama-benchy (0.3.5)
date: 2026-03-14 05:43:13 | latency mode: api
The results are slightly slower than the previous run.
The patches have already been published on GitHub.
I would be happy if they could be useful, even as a reference for your research.