GPT-OSS-120B MXFP4 on RTX PRO 6000 Blackwell Max-Q (SM120): full debug path, what was actually broken, and what finally worked

I want to share the full path I took to get GPT-OSS-120B with MXFP4 running on an NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, because I lost a lot of time debugging the wrong things, and maybe this saves someone else from doing the same.

First, the important conclusion: this does work on SM120. The GPU was consistently detected as compute capability 12.0 / SM120, both from PyTorch and from the vLLM startup path. So the final answer was not “Blackwell desktop/workstation cannot run this.” The answer was: my runtime path, environment, and container state were wrong.

I also want to say this clearly up front: without christopherowen’s work, and without the NVIDIA forum discussion and help from people there, I do not think I would have gotten this running. A big thank you especially to christopherowen, and also to the NVIDIA people and forum users who were discussing the 12.x, 12.xf, 12.xa behavior and the Blackwell feature targeting details. That context mattered a lot, because otherwise I would have kept assuming the GPU itself was the problem.

What I started with

My setup had an older runtime manager that was starting the model in Docker and exposing it as llama3 for compatibility with existing workers and websites. That part was intentional — I did not want to rewrite five different site worker stacks just to rename the model. The old manager was launching Docker with parameters like:

  • FLASHINFER_CUDA_ARCH_LIST=12.0

  • VLLM_MXFP4_BACKEND=marlin

  • --mxfp4-backend MARLIN

  • --attention-backend TRITON_ATTN

  • --gpu-memory-utilization 0.80

  • --max-num-seqs 200

That turned out to be a big part of the problem, because it meant I was not actually testing the cleaner runtime path I thought I was testing.

First major failure mode

With the original setup, the model could load most of the way, but then the engine died when JIT-building a quantization-related kernel. The key PTXAS failure was:

Feature 'cvt.e2m1x2.f32' not supported on .target 'sm_120'

That error was critical because it showed that the problem was not “model too big,” “BW unsupported,” or “MXFP4 impossible on this card.” It showed that the build/runtime path was ending up on a plain sm_120 target for a path that expected more. That was the first strong clue that the issue was really about architecture targeting and toolchain/runtime behavior, not raw hardware support.

Second independent failure mode

After that, I moved to a clean image and started testing more directly. Then I hit a different problem: if I used a multi-value FlashInfer architecture string like:

FLASHINFER_CUDA_ARCH_LIST=12.0f;12.1a

then flashinfer failed already on import, before even reaching model startup. The error was:

ValueError: too many values to unpack (expected 2)

That told me there was another problem entirely: in this setup, FlashInfer’s parsing logic did not like that multi-architecture string.

So I had two different problems:

  1. 12.0 let things start further, but eventually led to the bad sm_120 JIT/PTXAS path.

  2. 12.0f;12.1a failed even earlier, during FlashInfer import.

What finally unlocked the setup

The turning point was to stop thinking in terms of “generic Blackwell list” and instead force a single architecture target for my exact card, which is SM120.

The key runtime setting was:

export FLASHINFER_CUDA_ARCH_LIST=12.0f

Once I did that in a clean test container, flashinfer imported correctly, and both of these worked on GPU:

  • flashinfer.mxfp4_quantize

  • flashinfer.mxfp8_quantize

That was the point where it became obvious that the hardware path itself was fine. The Blackwell card was not the blocker. The problem was how I was driving the stack.

Why I stopped patching the old setup

At that point it became clear that my original environment was too contaminated to trust:

  • old containers were still around

  • port 8000 was still being held by stale Docker proxies

  • I had old manager code starting one runtime shape

  • I had a repo on disk suggesting another runtime shape

  • and I had already manually edited some files the previous day

So instead of continuing to patch that mess, I rebuilt everything from scratch in a clean new folder and gave it a new Docker image name, specifically so it would not get mixed up with the broken one.

Clean rebuild approach

I cloned the repo into a fresh directory, changed only one important line in the Dockerfile so that FLASHINFER_CUDA_ARCH_LIST became 12.0f instead of a list, and then built a fresh image with a fresh tag.

This was the clean rebuild process:

cd /home/emil
rm -rf /home/emil/bw-mxfp4-clean
mkdir -p /home/emil/bw-mxfp4-clean
cd /home/emil/bw-mxfp4-clean

git clone https://github.com/christopherowen/spark-vllm-mxfp4-docker.git .

cp Dockerfile Dockerfile.ORIG
sed -i 's/ENV FLASHINFER_CUDA_ARCH_LIST=.*/ENV FLASHINFER_CUDA_ARCH_LIST="12.0f"/' Dockerfile

sudo docker build -t vllm-mxfp4-bw-clean .

I deliberately used a new image name so I would never again confuse the new clean image with the older broken image.

Clean runtime test

Then I started a clean interactive container with the actual model directory and Hugging Face cache mounted. That part is important, because without those mounts you can get misleading model-path errors and think you are debugging MXFP4 when you are really just missing model files.

sudo docker run --rm -it \
  --gpus '"device=0"' \
  --ipc=host \
  -p 8000:8000 \
  -v /home/emil/.cache/huggingface:/root/.cache/huggingface \
  -v /home/emil/models:/models \
  --entrypoint bash \
  vllm-mxfp4-bw-clean

Inside that clean container, I used:

export FLASHINFER_CUDA_ARCH_LIST=12.0f
export PYTHONPATH=/workspace/flashinfer:/workspace/vllm
export VLLM_FASTSAFETENSORS_NOGDS=1
export FLASHINFER_LOGLEVEL=0

and then started the server with:

vllm serve /models/gpt-oss-120b \
  --host 0.0.0.0 \
  --port 8000 \
  --tokenizer openai/gpt-oss-120b \
  --served-model-name llama3 \
  --quantization mxfp4 \
  --mxfp4-backend CUTLASS \
  --mxfp4-layers moe,qkv,o,lm_head \
  --attention-backend FLASHINFER \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.70 \
  --max-model-len 8192 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --load-format fastsafetensors

What happened with the working config

With that configuration:

  • the model loaded successfully

  • lm_head quantization ran

  • vLLM compiled and initialized

  • KV cache was created

  • graph capture completed

  • the API server started successfully on port 8000

  • /health returned 200 OK

That was the first clean proof that GPT-OSS-120B MXFP4 was actually running on my SM120 Blackwell card.

What was not the main problem

There were still warnings, for example:

  • Triton warning about No module named 'triton_kernels.routing'

  • warnings around local path vs repo-id behavior when the model path was not mounted in earlier tests

But those were not the main reason the real run was failing. The real blockers were:

  1. stale Docker/container reuse

  2. broken older manager runtime parameters

  3. FLASHINFER_CUDA_ARCH_LIST=12.0 falling into the wrong JIT/PTXAS path

  4. FLASHINFER_CUDA_ARCH_LIST=12.0f;12.1a breaking FlashInfer import parsing

  5. testing with dirty old containers and reused ports instead of rebuilding cleanly

Final integration step

Once the clean runtime worked, I updated my rig_manager.py so that externally everything still looks like llama3, because that keeps my existing websites and workers happy, but internally the manager now launches the clean Blackwell image with the correct SM120 settings.

That let me keep compatibility while replacing the broken old runtime underneath. The old manager was hardcoded to the wrong values, which is why I kept recreating the same bad setup until I rewrote it.

Final practical advice

If someone else with an SM120 Blackwell card is stuck, my advice is:

Do not keep patching an old container.
Do not assume “Blackwell just cannot do it.”
Build a fresh image under a new name.
Force a single FlashInfer arch for your card:
FLASHINFER_CUDA_ARCH_LIST=12.0f
Mount your model path and Hugging Face cache explicitly.
Use the clean runtime path that actually worked for me:
CUTLASS + FLASHINFER, not the older MARLIN + TRITON_ATTN path I had in my original manager.
And clear stale containers and port bindings before each serious test.

Thanks

Again, big thanks to christopherowen for the actual repo/work that made this possible, and thanks to the NVIDIA forum people — especially the engineers and users discussing the 12.0f / 12.1a / 12.1f behavior and Blackwell feature targeting. Without that discussion, I would probably still be wasting time assuming the GPU itself was the issue.

Load test and result

After fixing the runtime path, I ran a larger native FP4 throughput test on the clean Blackwell setup.

Runtime config used:

  • FLASHINFER_CUDA_ARCH_LIST=12.0f
  • mxfp4-backend = CUTLASS
  • attention-backend = FLASHINFER
  • kv-cache-dtype = fp8
  • gpu-memory-utilization = 0.85
  • max-model-len = 8192
  • max-num-seqs = 500
  • max-num-batched-tokens = 8192

Workload:

  • 500 article generations
  • target length ~1500 tokens each

Observed result:

  • Total time: 150.1s
  • Total tokens: 694,906
  • Throughput: 4630 tok/s
  • Estimated articles/hour: 11,993
  • Successful requests: 500/500
  • Errors: 0

During the run, the server handled the load without queue buildup:

  • Waiting: 0 reqs
  • hundreds of concurrent running requests
  • stable prefix cache hit rate
  • no crash, no deadlock, no runtime failure

I also saw generation throughput peaks above 5k tok/s during the run, with the final aggregate average landing at about 4630 tok/s.

For me, that was the real proof point: this was not just a “server boots successfully” demo. It was a stable high-throughput batch inference run on SM120 Blackwell with GPT-OSS-120B MXFP4.

I do like this approach, I was running the @eugr recipe with really good success, but seems MXFP4 is now broken in the newest vLLM-dev release. Maybe this can help to get his GPT-OSS-120B recipe fixed. Which is currently :

# Recipe: OpenAI GPT-OSS 120B
# OpenAI's open source 120B MoE model with MXFP4 quantization support

recipe_version: "1"
name: OpenAI GPT-OSS 120B
description: vLLM serving openai/gpt-oss-120b with MXFP4 quantization and FlashInfer

# HuggingFace model to download (optional, for --download-model)
model: openai/gpt-oss-120b

# Container image to use
container: vllm-node-mxfp4

# Build arguments for build-and-copy.sh
build_args:
  - --exp-mxfp4

# No mods required for this model
mods: []

# Default settings (can be overridden via CLI)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.70
  max_num_batched_tokens: 8192

# Environment variables to set in the container
env:
  VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8: "1"

# The vLLM serve command template
# Uses MXFP4 quantization for memory efficiency
command: |
  vllm serve openai/gpt-oss-120b \
      --tool-call-parser openai \
      --reasoning-parser openai_gptoss \
      --enable-auto-tool-choice \
      --tensor-parallel-size {tensor_parallel} \
      --distributed-executor-backend ray \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --enable-prefix-caching \
      --load-format fastsafetensors \
      --quantization mxfp4 \
      --mxfp4-backend CUTLASS \
      --mxfp4-layers moe,qkv,o,lm_head \
      --attention-backend FLASHINFER \
      --kv-cache-dtype fp8 \
      --max-num-batched-tokens {max_num_batched_tokens} \
      --host {host} \
      --port {port}

have a look at this PR Pin nvidia-nvshmem-cu13 to <3.6 in Dockerfile.mxfp4 by ageev · Pull Request #152 · eugr/spark-vllm-docker · GitHub Falling back to pre-March24 NVSHMEM release fixed all my issues with mxfp4 container.

NOB question… how did you go about rolling back NVSHMEM ?

just two lines to replace in dockerfile.mxfp4. Search for “nvidia-nvshmem-cu13” and replace with “nvidia-nvshmem-cu13<3.6”. Lines 104 and 275. Then rebuild the container.

I did hit crashes at first under real load, specifically CUDA illegal memory access in the vLLM engine.

On my SM120 Blackwell card, the first key fix was forcing:

FLASHINFER_CUDA_ARCH_LIST=12.0f

Using 12.0 led me into the bad sm_120 path, while a multi-arch value like 12.0f;12.1a broke FlashInfer import in my environment.

The other important change on my side was adding:

--enforce-eager

That resolved the runtime crash I was seeing in practice.

So in my case, the setup became stable only after:

  1. rebuilding from a clean image,
  2. forcing FLASHINFER_CUDA_ARCH_LIST=12.0f,
  3. and adding --enforce-eager.

Without those changes, I was able to get the model to start, but under real traffic I would eventually hit engine crashes.

Now, MY TEST:
I tested both CUTLASS native FP4 and Marlin on SM120.

What I found is that CUTLASS was not universally faster.
For single requests they were basically tied (~190 tok/s).
CUTLASS won clearly in my synthetic batch tests:

  • 1540 vs 937 tok/s on 100 × 200-token batch
  • 4630 vs 3489 tok/s on 500 × 1500-token batch

However, Marlin was faster in my real production article pipeline:

  • ~23 min for 100 articles with Marlin
  • ~34 min with CUTLASS

So for me the real takeaway was:
CUTLASS looked better for API/batch-style synthetic serving, while Marlin performed better on my actual long-context production workflow.

Nice, I’ll test and merge if successful.

But in the long run we’ll need a more robust replacement - hopefully native flashinfer will get there, as this MXFP4 fork is not maintained by the original author anymore.

I can second that this solution works, GPT-OSS-120B MXFP4 rebuilt with ./run-recipe.sh openai-gpt-oss-120b --solo --setup --force-build and stable again.