I want to share the full path I took to get GPT-OSS-120B with MXFP4 running on an NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, because I lost a lot of time debugging the wrong things, and maybe this saves someone else from doing the same.
First, the important conclusion: this does work on SM120. The GPU was consistently detected as compute capability 12.0 / SM120, both from PyTorch and from the vLLM startup path. So the final answer was not “Blackwell desktop/workstation cannot run this.” The answer was: my runtime path, environment, and container state were wrong.
I also want to say this clearly up front: without christopherowen’s work, and without the NVIDIA forum discussion and help from people there, I do not think I would have gotten this running. A big thank you especially to christopherowen, and also to the NVIDIA people and forum users who were discussing the 12.x, 12.xf, 12.xa behavior and the Blackwell feature targeting details. That context mattered a lot, because otherwise I would have kept assuming the GPU itself was the problem.
What I started with
My setup had an older runtime manager that was starting the model in Docker and exposing it as llama3 for compatibility with existing workers and websites. That part was intentional — I did not want to rewrite five different site worker stacks just to rename the model. The old manager was launching Docker with parameters like:
-
FLASHINFER_CUDA_ARCH_LIST=12.0 -
VLLM_MXFP4_BACKEND=marlin -
--mxfp4-backend MARLIN -
--attention-backend TRITON_ATTN -
--gpu-memory-utilization 0.80 -
--max-num-seqs 200
That turned out to be a big part of the problem, because it meant I was not actually testing the cleaner runtime path I thought I was testing.
First major failure mode
With the original setup, the model could load most of the way, but then the engine died when JIT-building a quantization-related kernel. The key PTXAS failure was:
Feature 'cvt.e2m1x2.f32' not supported on .target 'sm_120'
That error was critical because it showed that the problem was not “model too big,” “BW unsupported,” or “MXFP4 impossible on this card.” It showed that the build/runtime path was ending up on a plain sm_120 target for a path that expected more. That was the first strong clue that the issue was really about architecture targeting and toolchain/runtime behavior, not raw hardware support.
Second independent failure mode
After that, I moved to a clean image and started testing more directly. Then I hit a different problem: if I used a multi-value FlashInfer architecture string like:
FLASHINFER_CUDA_ARCH_LIST=12.0f;12.1a
then flashinfer failed already on import, before even reaching model startup. The error was:
ValueError: too many values to unpack (expected 2)
That told me there was another problem entirely: in this setup, FlashInfer’s parsing logic did not like that multi-architecture string.
So I had two different problems:
-
12.0let things start further, but eventually led to the badsm_120JIT/PTXAS path. -
12.0f;12.1afailed even earlier, during FlashInfer import.
What finally unlocked the setup
The turning point was to stop thinking in terms of “generic Blackwell list” and instead force a single architecture target for my exact card, which is SM120.
The key runtime setting was:
export FLASHINFER_CUDA_ARCH_LIST=12.0f
Once I did that in a clean test container, flashinfer imported correctly, and both of these worked on GPU:
-
flashinfer.mxfp4_quantize -
flashinfer.mxfp8_quantize
That was the point where it became obvious that the hardware path itself was fine. The Blackwell card was not the blocker. The problem was how I was driving the stack.
Why I stopped patching the old setup
At that point it became clear that my original environment was too contaminated to trust:
-
old containers were still around
-
port 8000 was still being held by stale Docker proxies
-
I had old manager code starting one runtime shape
-
I had a repo on disk suggesting another runtime shape
-
and I had already manually edited some files the previous day
So instead of continuing to patch that mess, I rebuilt everything from scratch in a clean new folder and gave it a new Docker image name, specifically so it would not get mixed up with the broken one.
Clean rebuild approach
I cloned the repo into a fresh directory, changed only one important line in the Dockerfile so that FLASHINFER_CUDA_ARCH_LIST became 12.0f instead of a list, and then built a fresh image with a fresh tag.
This was the clean rebuild process:
cd /home/emil
rm -rf /home/emil/bw-mxfp4-clean
mkdir -p /home/emil/bw-mxfp4-clean
cd /home/emil/bw-mxfp4-clean
git clone https://github.com/christopherowen/spark-vllm-mxfp4-docker.git .
cp Dockerfile Dockerfile.ORIG
sed -i 's/ENV FLASHINFER_CUDA_ARCH_LIST=.*/ENV FLASHINFER_CUDA_ARCH_LIST="12.0f"/' Dockerfile
sudo docker build -t vllm-mxfp4-bw-clean .
I deliberately used a new image name so I would never again confuse the new clean image with the older broken image.
Clean runtime test
Then I started a clean interactive container with the actual model directory and Hugging Face cache mounted. That part is important, because without those mounts you can get misleading model-path errors and think you are debugging MXFP4 when you are really just missing model files.
sudo docker run --rm -it \
--gpus '"device=0"' \
--ipc=host \
-p 8000:8000 \
-v /home/emil/.cache/huggingface:/root/.cache/huggingface \
-v /home/emil/models:/models \
--entrypoint bash \
vllm-mxfp4-bw-clean
Inside that clean container, I used:
export FLASHINFER_CUDA_ARCH_LIST=12.0f
export PYTHONPATH=/workspace/flashinfer:/workspace/vllm
export VLLM_FASTSAFETENSORS_NOGDS=1
export FLASHINFER_LOGLEVEL=0
and then started the server with:
vllm serve /models/gpt-oss-120b \
--host 0.0.0.0 \
--port 8000 \
--tokenizer openai/gpt-oss-120b \
--served-model-name llama3 \
--quantization mxfp4 \
--mxfp4-backend CUTLASS \
--mxfp4-layers moe,qkv,o,lm_head \
--attention-backend FLASHINFER \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.70 \
--max-model-len 8192 \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--enable-prefix-caching \
--load-format fastsafetensors
What happened with the working config
With that configuration:
-
the model loaded successfully
-
lm_headquantization ran -
vLLM compiled and initialized
-
KV cache was created
-
graph capture completed
-
the API server started successfully on port 8000
-
/healthreturned 200 OK
That was the first clean proof that GPT-OSS-120B MXFP4 was actually running on my SM120 Blackwell card.
What was not the main problem
There were still warnings, for example:
-
Triton warning about
No module named 'triton_kernels.routing' -
warnings around local path vs repo-id behavior when the model path was not mounted in earlier tests
But those were not the main reason the real run was failing. The real blockers were:
-
stale Docker/container reuse
-
broken older manager runtime parameters
-
FLASHINFER_CUDA_ARCH_LIST=12.0falling into the wrong JIT/PTXAS path -
FLASHINFER_CUDA_ARCH_LIST=12.0f;12.1abreaking FlashInfer import parsing -
testing with dirty old containers and reused ports instead of rebuilding cleanly
Final integration step
Once the clean runtime worked, I updated my rig_manager.py so that externally everything still looks like llama3, because that keeps my existing websites and workers happy, but internally the manager now launches the clean Blackwell image with the correct SM120 settings.
That let me keep compatibility while replacing the broken old runtime underneath. The old manager was hardcoded to the wrong values, which is why I kept recreating the same bad setup until I rewrote it.
Final practical advice
If someone else with an SM120 Blackwell card is stuck, my advice is:
Do not keep patching an old container.
Do not assume “Blackwell just cannot do it.”
Build a fresh image under a new name.
Force a single FlashInfer arch for your card:
FLASHINFER_CUDA_ARCH_LIST=12.0f
Mount your model path and Hugging Face cache explicitly.
Use the clean runtime path that actually worked for me:
CUTLASS + FLASHINFER, not the older MARLIN + TRITON_ATTN path I had in my original manager.
And clear stale containers and port bindings before each serious test.
Thanks
Again, big thanks to christopherowen for the actual repo/work that made this possible, and thanks to the NVIDIA forum people — especially the engineers and users discussing the 12.0f / 12.1a / 12.1f behavior and Blackwell feature targeting. Without that discussion, I would probably still be wasting time assuming the GPU itself was the issue.