vLLM on GB10: gpt-oss-120b MXFP4 slower than SGLang/llama.cpp... what’s missing?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
21

Announcing: vLLM + native MXFP4 for gpt-oss-120b on DGX Spark (SM121/GB10) — reproducible Docker setup

I’m sharing a repo that packages a working, reproducible Docker environment for running OpenAI’s gpt-oss-120b (MXFP4 weights) on NVIDIA DGX Spark (SM121 / GB10) while moving towards the intended native Blackwell FP4 paths.

Repo:

Why this exists

Getting “real” MXFP4 performance on SM121 has a few sharp edges (toolchain + arch flags + backend selection + it didn’t work before). This repo is meant to make it easy for other Spark owners to:

  • stand up a working server quickly
  • reproduce performance results
  • help validate correctness/perf and upstream the right pieces

There’s more work to do, and I’ve outlined what is done and what more there is to do in the repo.

I’m very interested in either feedback or contributions.

22

I think I may have worked around the issue with caching in the repro Dockerfile. I think I must have been triggering the rebuilds from all the development I was doing!

23

Hardware Features Enabled by 12.1f:

  1. Block-scaled MMA instructions - mma.sync.aligned.kind::mxf4.block_scale.m16n8k64 - the core MXFP4 tensor core operation

  2. FP4 ldmatrix with format conversion - ldmatrix.b8x16.b4x16_p64 - efficient FP4 data loading

  3. tcgen05 tensor core generation - Blackwell’s new tensor core with asymmetric tile requirements (M≥64, N≥8)

Without 12.1f (using 12.1a or plain 12.1):

  • Compiler rejects block-scaled MMA instructions

  • Compiler rejects FP4 ldmatrix instructions

  • Falls back to generic code that doesn’t use MXFP4 hardware

The a suffix (architecture-specific) explicitly excludes these newer features on SM121. The f suffix (family mode) enables the full Blackwell FP4/FP8 instruction set.

I documented some of the research here: spark-vllm-mxfp4-docker/docs/reference/SM121_TECHNICAL_GUIDE.md at main · christopherowen/spark-vllm-mxfp4-docker · GitHub

24

Interesting, I guess I will need to make 12.1f a new default in the docker builds… @johnny_nv - any comments on that? Seems weird that “family” instruction set is wider than specific arch instruction set.

25

that not should be true.

In fact 12.1a use full features from the chip.

For example, for 10.3a (gb300) if we put 10.0f we lose some features in flashinfer or sglang so the execution can break

F uses major family features.

without letter only cuda kernels.

I am on vacation now, but i can ingestigate it in the following weeks. Maybe they are filtering something:

12.a=12.f false in the utils file code

26

Yeah, that’s what I thought. Likely a bug in flashinfer then.

27

I’ve ran into at least 1 other bug with nvcc and pxtas while developing on the DGX. I wouldn’t be surprised to learn this was a preexisting one or one that I introduced. I will try to revisit and produce a repro.

There is evidence of this in the flashinfer repo itself: Code search results · GitHub

I am currently using 12.1f (as opposed to 12.1a, 12.0f, or 12.0a).

28

Just validated the MXFP4 + FlashInfer stack for GPT-OSS-120B on a DGX Spark (SM121/GB10).
It works - and it works fast. We’re seeing the expected ~59 tok/s decode on long sequences, with stable behavior up to 131k context.

Huge thanks for making this possible - this project is absolutely fantastic.
Would be great to refresh the MXFP4 containers on NVIDIA’s side, and also ping @eugr for a synced update.

Appreciate the effort — impressive work!

dgx@gx10-4323:~$ docker exec -it vllm-gptoss120b
llama-benchy
–base-url http://localhost:30000/v1
–model gpt-oss-120b
–tokenizer /model
–pp 512 2048 8192 16384
–tg 32 128
–runs 5
llama-benchy (0.1.1)
Date: 2026-01-18 18:23:19
Benchmarking model: gpt-oss-120b at http://localhost:30000/v1
Loading text from cache: /root/.cache/llama-benchy/cc6a0b5782734ee3b9069aa3b64cc62c.txt
Total tokens available in text corpus: 140865
Warming up…
Warmup (User only) complete. Delta: 65 tokens (Server: 86, Local: 21)
Warmup (System+Empty) complete. Delta: 69 tokens (Server: 90, Local: 21)
Measuring latency using mode: api…
Average latency (api): 1.23 ms
Running test: pp=512, tg=32, depth=0
Running test: pp=512, tg=128, depth=0
Running test: pp=2048, tg=32, depth=0
Running test: pp=2048, tg=128, depth=0
Running test: pp=8192, tg=32, depth=0
Running test: pp=8192, tg=128, depth=0
Running test: pp=16384, tg=32, depth=0
Running test: pp=16384, tg=128, depth=0

model test t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
gpt-oss-120b pp512 2115.76 ± 32.92 243.28 ± 3.69 242.05 ± 3.69 295.25 ± 4.26
gpt-oss-120b tg32 59.36 ± 0.51
gpt-oss-120b pp512 2097.99 ± 27.01 245.32 ± 3.12 244.08 ± 3.12 296.46 ± 3.08
gpt-oss-120b tg128 59.70 ± 0.05
gpt-oss-120b pp2048 4771.75 ± 39.70 430.45 ± 3.58 429.22 ± 3.58 482.67 ± 3.71
gpt-oss-120b tg32 58.64 ± 0.41
gpt-oss-120b pp2048 4716.49 ± 61.54 435.53 ± 5.72 434.30 ± 5.72 488.15 ± 4.86
gpt-oss-120b tg128 59.22 ± 0.07
gpt-oss-120b pp8192 6626.55 ± 9.41 1237.47 ± 1.75 1236.24 ± 1.75 1290.59 ± 2.04
gpt-oss-120b tg32 56.77 ± 0.49
gpt-oss-120b pp8192 6621.92 ± 15.85 1238.34 ± 2.96 1237.11 ± 2.96 1292.36 ± 3.57
gpt-oss-120b tg128 57.27 ± 0.13
gpt-oss-120b pp16384 5920.72 ± 15.52 2768.48 ± 7.25 2767.25 ± 7.25 2824.37 ± 6.84
gpt-oss-120b tg32 54.82 ± 0.39
gpt-oss-120b pp16384 5900.69 ± 9.37 2777.87 ± 4.41 2776.63 ± 4.41 2833.64 ± 4.71
gpt-oss-120b tg128 55.12 ± 0.08
29

This is great stuff. Unfortunately, having an issue when building per instructions. I should say – build is fine, but running with Docker or Docker compose per provided examples fail. Stack as follows:

(APIServer pid=1) INFO 01-18 19:40:15 [config.py:307] Overriding max cuda graph capture size to 1024 for performance.

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main

    exitcode = _main(fd, parent_sentinel)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 131, in _main

    prepare(preparation_data)

  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 246, in prepare

    _fixup_main_from_path(data['init_main_from_path'])

  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path

    main_content = runpy.run_path(main_path,

                   ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "<frozen runpy>", line 280, in run_path

  File "/usr/lib/python3.12/pkgutil.py", line 170, in <module>

    iter_importer_modules.register(

  File "/usr/lib/python3.12/functools.py", line 892, in register

    if _is_union_type(cls):

       ^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/functools.py", line 842, in _is_union_type

    from typing import get_origin, Union

  File "/usr/local/lib/python3.12/dist-packages/cv2/typing/__init__.py", line 61, in <module>

    import numpy

  File "/usr/local/lib/python3.12/dist-packages/numpy/__init__.py", line 181, in <module>

    from . import lib

  File "/usr/local/lib/python3.12/dist-packages/numpy/lib/__init__.py", line 23, in <module>

    from . import _index_tricks_impl

  File "/usr/local/lib/python3.12/dist-packages/numpy/lib/_index_tricks_impl.py", line 12, in <module>

    import numpy.matrixlib as matrixlib

  File "/usr/local/lib/python3.12/dist-packages/numpy/matrixlib/__init__.py", line 4, in <module>

    from . import defmatrix

  File "/usr/local/lib/python3.12/dist-packages/numpy/matrixlib/defmatrix.py", line 12, in <module>

    from numpy.linalg import matrix_power

  File "/usr/local/lib/python3.12/dist-packages/numpy/linalg/__init__.py", line 88, in <module>

    from . import _linalg

  File "/usr/local/lib/python3.12/dist-packages/numpy/linalg/_linalg.py", line 22, in <module>

    from typing import NamedTuple, Any

ImportError: cannot import name 'NamedTuple' from partially initialized module 'typing' (most likely due to a circular import) (/usr/local/lib/python3.12/dist-packages/cv2/typing/__init__.py)

(APIServer pid=1) Traceback (most recent call last):

(APIServer pid=1)   File "/usr/local/bin/vllm", line 10, in <module>

(APIServer pid=1)     sys.exit(main())

(APIServer pid=1)              ^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/entrypoints/cli/main.py", line 73, in main

(APIServer pid=1)     args.dispatch_function(args)

(APIServer pid=1)   File "/workspace/vllm/vllm/entrypoints/cli/serve.py", line 60, in cmd

(APIServer pid=1)     uvloop.run(run_server(args))

(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run

(APIServer pid=1)     return __asyncio.run(

(APIServer pid=1)            ^^^^^^^^^^^^^^

(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run

(APIServer pid=1)     return runner.run(main)

(APIServer pid=1)            ^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run

(APIServer pid=1)     return self._loop.run_until_complete(task)

(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete

(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper

(APIServer pid=1)     return await main

(APIServer pid=1)            ^^^^^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1325, in run_server

(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)

(APIServer pid=1)   File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1344, in run_server_worker

(APIServer pid=1)     async with build_async_engine_client(

(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__

(APIServer pid=1)     return await anext(self.gen)

(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 171, in build_async_engine_client

(APIServer pid=1)     async with build_async_engine_client_from_engine_args(

(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__

(APIServer pid=1)     return await anext(self.gen)

(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 212, in build_async_engine_client_from_engine_args

(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(

(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config

(APIServer pid=1)     return cls(

(APIServer pid=1)            ^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 134, in __init__

(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(

(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client

(APIServer pid=1)     return AsyncMPClient(*client_args)

(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/core_client.py", line 824, in __init__

(APIServer pid=1)     super().__init__(

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/core_client.py", line 479, in __init__

(APIServer pid=1)     with launch_core_engines(vllm_config, executor_class, log_stats) as (

(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__

(APIServer pid=1)     next(self.gen)

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/utils.py", line 921, in launch_core_engines

(APIServer pid=1)     wait_for_engine_startup(

(APIServer pid=1)   File "/workspace/vllm/vllm/v1/engine/utils.py", line 980, in wait_for_engine_startup

(APIServer pid=1)     raise RuntimeError(

(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1}

Any clue or anyone with a similar problem?

30

I’ll be incorporating this into my repo as a separate build flag (until it’s all merged upstream).

31

Hi Christopher, thanks for the validation and the great work!

I can confirm your findings on the DGX Spark (GB10). The single-user latency (approx. 59 tok/s generation) is exactly what we initially measured as well. It’s rock stable even at long contexts.

However, we found that this hardware (GB10 with MXFP4) has a massive headroom when you push it with high concurrency, which is a game-changer for high-throughput RAG scenarios (processing many small chunks in parallel).

Unlock: High-Concurrency RAG

We realized the GPU was barely sweating at standard batch sizes (1-2% KV cache usage). By tuning vLLM to aggressively fill the pipeline, we achieved throughputs I’ve never seen on a single card before.

Our “Mini Chunk” Stress Test Setup:

  • Scenario: High-speed RAG (processing hundreds of small context chunks simultaneously).
  • Concurrency: 1,024 concurrent requests.
  • Tuning: max-num-seqs 1024, max-num-batched-tokens 32768.

The Results (Single GB10): While single-user speed stays around ~60 tok/s, the aggregate system throughput explodes:

  • Prompt Processing (Reading): Peak ~9,840 tokens/s
  • Generation (Writing): Peak ~2,350 tokens/s

Server Log Snapshot:

Plaintext(APIServer pid=1) Engine 000: Avg prompt throughput: 9840.0 tokens/s, Avg generation throughput: 2046.9 tokens/s, Running: 990 reqs (APIServer pid=1) Engine 000: Avg prompt throughput: 1768.3 tokens/s, Avg generation throughput: 2323.8 tokens/s, Running: 1017 reqs

Conclusion: For anyone building RAG pipelines: This setup is a beast. You can process massive amounts of context in seconds if you parallelize your chunks. The GB10 eats concurrency for breakfast.

Thanks again for the initial stack validation!

32

what steps did you perform?

33

What changes did you make to your config?

34

I wouldn’t necessarily trust vLLM log. It often produces the numbers that don’t match the reality (e.g. when benchmarking on the client side). I would doublecheck using vllm bench serve and run vllm with --no-enable-prefix-caching flag just to make sure there are no duplicate requests (although I believe they fixed it):

vllm bench serve   --backend vllm   --model openai/gpt-oss-120b   --endpoint /v1/completions   --dataset-name sharegpt   --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json   --num-prompts 1024   --port 8000

(I’d recommend llama-benchy, but it doesn’t support concurrency yet).

35

Here is the measurement with a random input
stor@gx10-9eba:~/compose$ docker exec -it vllm_gb10_server vllm bench serve --backend vllm --model openai/gpt-oss-120b --endpoint /v1/completions --dataset-name random --num-prompts 1024 --port 8355

============ Serving Benchmark Result ============
Successful requests:                     1024
Failed requests:                         0
Benchmark duration (s):                  296.73
Total input tokens:                      1048576
Total generated tokens:                  131072
Request throughput (req/s):              3.45
Output token throughput (tok/s):         441.71
Peak output token throughput (tok/s):    3072.00
Peak concurrent requests:                1024.00
Total token throughput (tok/s):          3975.43
---------------Time to First Token----------------
Mean TTFT (ms):                          139312.19
Median TTFT (ms):                        133535.00
P99 TTFT (ms):                           235001.16
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1219.93
Median TPOT (ms):                        1265.69
P99 TPOT (ms):                           2010.25
---------------Inter-token Latency----------------
Mean ITL (ms):                           1220.01
Median ITL (ms):                         497.47
P99 ITL (ms):                            14742.24
==================================================

In my setup I have about 350k token from the database, mixed dense and bm25 results. the bm25 results are extended to the whole document, the dense ones, just the chunked in size up to 2k. In the first step I summarize all long documents 80-100. In the second step, i have all dense and summarized results as a list being chunked to fit the vectors to less then 4k (sometimes 8k). this 30-80 requests are then processed to a new list (about 2k-3k), which is feed through the LLM, for the final result. In all steps, except the last I need highest possible throughput. in the last step i have now 59 tps throughput. Here are parts of the docker-compose:

# Settings
      - VLLM_USE_V1=0
      #- VLLM_ATTENTION_BACKEND=FLASHINFER  in command
      - VLLM_WORKER_MULTIPROC_METHOD=spawn
      - VLLM_NO_USAGE_STATS=1
      - SAGEMAKER_PROGRAM=null
      - LOG_LEVEL=INFO

    command: >-
      python3 -m vllm.entrypoints.openai.api_server
      --model openai/gpt-oss-120b
      --quantization mxfp4
      --mxfp4-backend CUTLASS
      --mxfp4-layers moe,qkv,o,lm_head 
      --attention-backend FLASHINFER 
      --tensor-parallel-size 1 
      --kv-cache-dtype fp8
      --max-num-seqs 1024
      --max-num-batched-tokens 65535 
      --max-model-len 4096 
      --gpu-memory-utilization 0.85  
      --enable-prefix-caching 
      --load-format fastsafetensors          
      --port 8355
      --host 0.0.0.0
      --disable-log-requests

for production runs, I increase
–max-model-len 4096 to 8192 or 16384. and take
–max-num-batched-tokens 65535 to 32768.
Depending in config I get max to

Output token throughput (tok/s):         514.49
Peak output token throughput (tok/s):    2941.00
Peak concurrent requests:                1024.00
Total token throughput (tok/s):          4630.40

or prefill up to:

vllm_gb10_server  | (APIServer pid=1) INFO 01-20 19:55:57 [loggers.py:257] Engine 000: Avg prompt throughput: 7380.6 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 36 reqs, Waiting: 8 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 1.4%
vllm_gb10_server  | (APIServer pid=1) INFO 01-20 19:56:07 [loggers.py:257] Engine 000: Avg prompt throughput: 12616.2 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 1.4%

So I can to through a 350k context window in about 3 minutes an get a high quality output (no chance in a 3 single runs!, quality wise)! No other test except with genini 3.0 was getting close.
Qwen3-Next.80B, nemotron or so have much lower quality, output (my most import ant metrik. I’m not time critical, but still, the faster, the better!

36

Looks like it was the same issue reported here: Circular Import · Issue #12 · eugr/spark-vllm-docker · GitHub

Going to try to pull a fresh image

37

These are nice numbers! Although I’ve had issues with vllm bench serve when it reused the prompts even with random dataset, so to be sure, I’d run vllm with --no-enable-prefix-caching.

38

I hit the same problem.

Have made a small touch to the Dockerfile and put it in the PR

39

Thank you for your contribution! I am eager to merge it.

I have made a few comments in your PR for your review, please.

40

BTW, I’ve incorporated your changes into my repository. I’ve added building flashinfer-cubin and flashinfer-jit-cache packages, so the cold startup should be improved. I tested it, and it worked fine, now just need to incorporate into the build script as a separate flag and I’ll release it.