I am currently working on successfully deploying vLLM 0.17.0 in a DGX Spark environment (GB10 / SM121) using MXFP4 quantization. I’ve put together a set of patches that fix SM121-specific issues and add BF16 → MXFP4 online quantization for attention layers and lm_head. Here are the results on a 2-node DGX Spark cluster (TP=2, single request, vllm bench serve with 1024 input / 128 output tokens):
Model
Vanilla
Patched
Speedup
Qwen3.5-35B-A3B
42.85 tok/s
70.68 tok/s
+65%
openai/gpt-oss-120b
51.82 tok/s
80.88 tok/s
+56%
The patches also enable TP=1 single-GPU inference, which vanilla vLLM cannot do on GB10 for MXFP4 models.
What’s included
BF16 → MXFP4 online quantization for MoE experts, attention (qkv_proj, o_proj), and lm_head via Marlin backend
SM121 device support for CUTLASS MoE kernels
Fix for weight_loader ndim mismatch with BF16 per-expert tensors
GDN (Gated Delta Net) Triton kernel fix for Qwen3.5
FlashInfer CUTLASS header bug fix
Installation
Takes just a few minutes — create a venv, pip install vLLM, apply two patch files, done. Full instructions and patches are on GitHub:
Thanks for the suggestion. I’m planning to submit a PR to CUTLASS for the SFA/SFB fix — it’s a straightforward copy-paste bug that should be easy to merge.
For the vLLM patches, the MXFP4 online quantization changes are fairly extensive and SM121-specific, so they would need significant refactoring to meet upstream code quality standards.
The GDN kernel change is more of a tuning adjustment for Qwen3.5’s dimension sizes rather than a bug fix, so it would need discussion with the FLA developers about the original design intent first.
As for the Triton patch, the explicit allocator setting via ContextVar appears to be an intentional design — the root issue is really about Ray’s thread-level ContextVar propagation.
So for now, everything except the CUTLASS fix will stay as out-of-tree patches.
I’ll test your patches and may include them as a mod in our community docker (I already have one for the triton allocator bug) if you don’t mind, as our current MXFP4 build is broken in the cluster setup.
Thanks! I’ve looked at @christopher_owen’s work. His patches extensively customize the CUTLASS kernels themselves, while mine improve the Marlin backend, so they don’t conflict. However, combining them would require unifying the quantization dispatch logic, which could be non-trivial.
Regarding the CUTLASS SFA/SFB fix — since @christopher_owen has already patched all three affected files (sm120, sm100, sm103), it would make sense for him to submit the upstream PR.
@eugr Feel free to include the patches in your community docker — happy to have them used.
Correction on TP=1 for gpt-oss-120b: After further testing, I found that TP=1produces malformed output for chat completions (Harmony protocol). Basic text completion works, but the structured reasoning tokens are corrupted — likely due to FP4 quantization errors affecting control token generation. The vllm bench serve benchmarks only measure throughput and don’t catch this. TP=2 remains unaffected. I’ve updated the README accordingly.
Update: TP=1 issue with gpt-oss-120b on SM121 has been fixed
I have identified and fixed the root cause of the malformed output when using TP=1.
The issue was a shared memory race condition in the Marlin MoE 256-thread kernel on SM121.
The Marlin MoE GEMM kernel has two thread configurations: a 128-thread configuration (thread_k=64, thread_n=128) and a 256-thread configuration (thread_k=128, thread_n=256). With gpt-oss-120b at TP=1, the w2 GEMM reduction dimension (intermediate size) N=2944 is large enough that the kernel selects the 256-thread configuration. However, on SM121, this 256-thread kernel suffers from a shared memory race condition that silently corrupts the w2 GEMM output, resulting in garbage data being generated.
With TP=2, the weight matrices are split across two GPUs, so N≈1472 — small enough that the kernel stays on the safe 128-thread configuration. This is why TP=2 worked correctly while TP=1 did not.
The vllm_all.patch also includes a fix for the Mxfp4Config.from_config() method, which was ignoring the modules_to_not_convert field from the model config. This caused attention layers and lm_head to be incorrectly FP4-quantized.
Hi team, thanks for the patch :) I applied it and I am using a single DGX configuration. It is pretty fast indeed, however I think it has some issues in handling the tools correctly. I got some glithces in the output, some randomic chinese words, and tool suddenly stop during mcp calls. I start vllm like this: export CUDA_HOME=/usr/local/cuda
First, download the latest patch from the GitHub repository and follow the instructions to execute the commands.
Run the program (with the --tensor-parallel-size 1 option) to verify that it functions correctly.
Once you’ve confirmed successful operation with this basic test, the next step is to incrementally add options one by one while identifying the root cause of the problem.
If you encounter unusual output during the basic test itself, this may indicate that the patch wasn’t applied correctly or that the README.md documentation lacks sufficient environment configuration instructions.
I tried running Qwen3.5-122B-A10B on 2x DGX Spark (TP=2), but it does not fit in memory.
The BF16 model weights are ~234GB (39 shards). With TP=2, each node needs to load ~117GB of BF16 weights before MXFP4 online quantization can convert them to ~35GB. Since each GB10 has only ~119GB usable unified memory, the loading process itself exceeds available memory and triggers OOM.
A pre-quantized checkpoint like Qwen/Qwen3.5-122B-A10B-FP8 (~127GB) could load successfully with --quantization fp8 (~64GB/node), but in that case the MXFP4 patches would not be used.
This model would likely work on TP=4 (4x DGX Spark) with MXFP4 — each node would only need ~59GB for BF16 loading, well within the 119GB limit. If anyone has a 4-node setup, I’d be curious to hear the results.