[SM121] 4 bugs causing ! output + gpt-oss-120B at 59 tok/s — full root cause analysis and working serve scripts

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi everyone,

I’ve been running NVFP4/MXFP4 models on a single DGX Spark (GB10, SM121, 128GB unified memory) for the past few weeks and wanted to share what I found — both the root causes of the !!!!! garbage output and working serve scripts for gpt-oss-120B (59 tok/s) and Qwen3.5-35B (59 tok/s).

Acknowledgments first: This work builds on eugr’s SM121 MXFP4 patches, christopher_owen’s SM121 technical research, and namake-taro’s vLLM fork which fixed the PTX and Marlin race bugs. What follows are additional issues I found when applying these patches to stock vLLM 0.17.1 and serving different model formats.

Part 1: Why any NVFP4 model outputs !!! on SM121

Four independent bugs, all need to be fixed. Fixing three out of four still gives garbage.

Bug 1: cutlass_fp4_supported() false positive

In nvfp4_utils.py, the function passes capability_int = 121 (for SM 12.1) to cutlass_scaled_mm_supports_fp4(). Since 121 exceeds any reasonable threshold, it returns True. But cutlass_scaled_mm_supports_fp4(device_id=0) correctly returns False on GB10. CUTLASS FP4 gets selected for all linear layers — and produces row-identical garbage on SM121.

The diagnostic signature: every element in an output row has the same wrong value (e.g., [-28.625, -28.625, -28.625, ...]). Not zeros, not random noise — same value repeated. This is how you confirm it’s CUTLASS FP4 and not something else.

Bug 2: CutlassExpertsFp4 matches SM121

is_device_capability_family(120) in cutlass_moe.py returns True for any SM12x, including SM121. MoE expert GEMMs also get the broken kernel. Important: VLLM_NVFP4_GEMM_BACKEND=marlin only affects linear layers. The MoE path ignores this env var entirely.

Fix:

python

# cutlass_moe.py
@staticmethod
def _supports_current_device() -> bool:
    p = current_platform
    cap = p.get_device_capability()
    if cap is not None and cap.major == 12 and cap.minor >= 1:
        return False  # SM121 produces garbage with CUTLASS FP4
    return p.is_cuda() and (
        p.is_device_capability_family(100)
        or p.is_device_capability_family(110)
        or p.is_device_capability_family(120)
    )

Bug 3: SupportsQuant missing on Qwen3.5 model class

This one is Qwen3.5-122B specific. The hybrid GDN (SSM) layers must stay BF16, but Qwen3_5ForConditionalGeneration doesn’t inherit SupportsQuant, so apply_vllm_mapper() is never called, and the exclude_modules list uses wrong-format weight names. All GDN layers get NVFP4-quantized silently.

Fix: class Qwen3_5ForConditionalGeneration(Qwen3VLForConditionalGeneration, IsHybrid, SupportsQuant):

Bug 4: PTX + Marlin race — already fixed in namake-taro’s fork and eugr’s patches.

Environment variables needed (all four):

bash

export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_MXFP4_USE_MARLIN=1
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_MARLIN_USE_ATOMIC_ADD=1

Full root cause analysis with detailed diagnostics: https://ai-muninn.com/en/blog/part1-why-your-dgx-spark-says-exclamation-marks

Part 2: gpt-oss-120B at 59 tok/s — 6 additional pitfalls

After fixing the NVFP4 bugs above, gpt-oss-120B has its own set of landmines:

1. eugr patch import path mismatch — mxfp4.py imports from quant_utils (eugr fork path), stock vLLM 0.17.1 uses nvfp4_utils. Server crashes on startup. One-line fix.

2. --enforce-eager in serve script — debug flag that disables CUDAGraph. 26 tok/s → 59 tok/s just by removing it. Check your scripts.

3. tiktoken vocab download on air-gapped machines — gpt-oss uses openai_harmony tokenizer. On internal networks, download fails. The cache uses SHA1 hashes as filenames (undocumented — had to read tiktoken-rs source):

bash

mkdir -p ~/models/tiktoken_cache
wget "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken" \
  -O ~/models/tiktoken_cache/fb374d419588a4632f3f557e76b4b70aebbca790
export TIKTOKEN_RS_CACHE_DIR=~/models/tiktoken_cache

(I saw someone asking about this exact error in the previous gpt-oss thread — this is the fix.)

4. content: null with reasoning parser — --reasoning-parser openai_gptoss puts everything in reasoning field. Standard clients get empty responses.

5. System message bypasses harmony encoding — vLLM PR #31607 (unmerged). Clients sending system prompts hit malformed token sequences. Requires patching serving_chat.py + setting VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS=1.

6. Wrong env var name (the hardest to find) — VLLM_NVFP4_GEMM_BACKEND does NOT exist in vLLM 0.17.1. It’s silently ignored. Correct var: VLLM_MXFP4_BACKEND=marlin. Without this, vLLM auto-selects CUTLASS_FP4, which causes repetition loops on SM121.

How to verify: Check startup log. You want to see:

[MXFP4] Using backend: marlin (VLLM_MXFP4_BACKEND=marlin)

Not:

[MXFP4] Auto-selected: CUTLASS_FP4 (vLLM native SM120 FP4 grouped GEMM for SM12x)

Working gpt-oss-120B serve script:

bash

#!/bin/bash
export VLLM_MXFP4_BACKEND=marlin
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export FLASHINFER_DISABLE_VERSION_CHECK=1
export TIKTOKEN_RS_CACHE_DIR=~/models/tiktoken_cache
export VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS=1

vllm serve ~/models/gpt-oss-120b \
  --served-model-name gpt-oss-120b --host 0.0.0.0 --port 8001 \
  --quantization mxfp4 --kv-cache-dtype fp8 \
  --max-model-len 131072 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 --attention-backend TRITON_ATTN \
  --moe-backend marlin

Result: 59 tok/s decode, 131K context, single GB10. Close to the 273 GB/s bandwidth ceiling.

Full writeup with all 6 bugs explained: https://ai-muninn.com/en/blog/part2-gpt-oss-120b-serve-script

Performance summary (single DGX Spark, 128GB)

Model Quantization Speed Max Context
Qwen3.5-35B (BF16→MXFP4 online) MXFP4 59 tok/s 200K
gpt-oss-120B MXFP4 57-59 tok/s 131K
Qwen3.5-122B NVFP4 Marlin W4A16 ~15 tok/s 200K

Note: both 35B and 120B can’t run simultaneously — each takes 65-75GB, total exceeds 128GB.

The 122B at 15 tok/s is Marlin W4A16 (bandwidth-bound). True W4A4 CUTLASS would be 2-3x faster but there’s no working SM121 kernel yet. Writeup on the 122B GDN bottleneck: https://ai-muninn.com/en/blog/part2-qwen-122b-14-toks-gdn-kernel-gap

Happy to answer questions. All detailed writeups are on my blog at https://ai-muninn.com — also available in Chinese (zh-TW).

2

Hello,

I don’t know what did you use to benchmark and which docker image you’ve used for vLLM, but I can run GPT-OSS-120B at 58-60 tokens/s on a single spark for months now.

Qwen 3.5 35B at FP8, not MXFP4 also runs at 52-55 tokens/s on single spark

And Intel/Qwen3.5-122B-A10B-int4-AutoRound runs at 28-29 tokens per second on a single Spark

It would be good for you to have created a mod with the patches to the image and a recipe or a command to replicate exactly your results. NVFP4 still has worse performance then int4-autoround on the Spark, all benchmarks point to it today. We’re still expecting better support for it by NVIDIA.

3

when i run my qwen3.5-35B (BF16 the memory was 110GB for some reason and my spark was super hot. waiting for NVFP4 and SM121 update for vllm