Qwen3.8-Flash-Next NVFP4 on 2x DGX Spark - recipe, benchmarks, and the patch it needs to load

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I spent a day getting the RadixArk NVFP4 quant of Qwen3.8-Flash-Next serving TP=2 across two DGX Sparks, and measured it against the FP8 build on the same rig. Sharing the recipe and the numbers, since the model card only documents an SGLang path and I could not find a vLLM one.

The checkpoint does not load as published. vLLM dies during weight loading with:

ValueError: There is no module or parameter named 'ngram_embedding.weight_scale'
in Qwen3_8FlashNextNGramEmbedding. The available parameters belonging to
ngram_embedding (VocabParallelEmbedding) are: {'ngram_embedding.weight'}

It is mixed precision: NVFP4 routed experts, but the PLE embedding tables are still FP8, 128 shards of F8_E4M3 [2500012, 160] plus one bf16 scalar scale. The quant config excludes the ple modules from NVFP4, so vLLM builds a plain VocabParallelEmbedding with no weight_scale slot while the bytes on disk are still FP8. The FP8 build has the identical 128 shard plus scale layout and loads fine, which is what shows it is the config path and not the tensors.

The gate is one line in models/qwen3_8_flash_next/nvidia/ple_layer.py:

def _get_ple_embedding_quant_method(quant_config, prefix):
    if not isinstance(quant_config, Fp8Config):
        return None
    ...
    return Qwen3_8FlashNextPLEFp8EmbeddingMethod()

Widening it so a ModelOpt config also returns Qwen3_8FlashNextPLEFp8EmbeddingMethod is enough, and it is safe because that class takes no constructor arguments and registers its own fp8 weight and bf16 scale. I bind mount the patched file over the image rather than baking it in. Do not work around it by deleting the stray scale, the shards really are F8_E4M3 and you would get silent corruption instead of an error. modelopt_mixed does not help either, it fails the same isinstance check.

My config:

vllm serve /model \
  --served-model-name qwen3.8-flash-next-nvfp4 \
  --tensor-parallel-size 2 --enable-expert-parallel \
  --gpu-memory-utilization 0.85 \
  --max-model-len 262144 \
  --max-num-seqs 64 \
  --kv-cache-dtype bfloat16 \
  --enforce-eager \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --distributed-executor-backend mp \
  --nnodes 2 --node-rank 0 --master-addr <fabric ip> --master-port 25400

Environment that matters:

VLLM_QWEN38FN_PLE_FP8=1
NCCL_MIN_NCHANNELS=4
NCCL_MAX_NCHANNELS=4
NCCL_SOCKET_IFNAME=<your CX7 interface>

enforce-eager is required, the full cudagraph modes wedge these boxes. bfloat16 KV is required, the QSA kernels declare supported_kv_cache_dtypes = [“auto”,“bfloat16”] and raise otherwise. The default NCCL channel count of 64 hangs channel init here. Also launch the container with --entrypoint bash, the image ENTRYPOINT is already vllm serve so a bare -c is eaten as --compilation-config.

Both ranks load at 64.06 GiB per node against 88.07 for FP8, and the engine reports:

GPU KV cache size: 2,228,932 tokens, Maximum concurrency for 262,144 tokens per request: 8.50x

FP8 on identical settings gives 607,890 tokens and 2.32x.

Concurrency ladder, 512 output tokens, both lanes at max-num-seqs 64, zero failures either side:

conc   FP8 agg   NVFP4 agg    delta
   1     23.02       29.61   +28.6%
   2     42.08       55.49   +31.9%
   4     61.90       92.42   +49.3%
   8     94.28      143.53   +52.2%
  16    145.84      212.09   +45.4%
  32    152.66      231.30   +51.5%
  64    176.75      257.59   +45.7%

Needle in a haystack, 5 needles:

context     FP8 ttft  NVFP4 ttft   FP8 dec  NVFP4 dec  needles
  4,096       3.6 s       3.1 s     37.24      50.95   5/5 both
 32,768      15.5 s      12.1 s     37.20      41.51   5/5 both
131,072      63.4 s      50.6 s     43.54      42.74   5/5 both
200,000      95.7 s      77.3 s     38.15      49.64   5/5 both
245,000     118.8 s      96.9 s     29.78      49.40   5/5 both

Both stop around 248K in practice. A 258K target fails with HTTP 400 because the rendered prompt reaches 261,583 tokens and leaves no room to generate inside the 262,144 window.

On agentic quality I ran a 40 instance SWE-bench Pro subset against each, same harness and same prompts: 36 of 40 for both. NVFP4 finished in 2h30m against 3h21m, and they fail different instances, with 2 in common.

One thing that surprised me: the KV difference does not buy longer context, both cap at max-model-len. It buys concurrency at long context, 8.50 simultaneous full length requests against 2.32.

Thanks to @eugr, whose container work most of us are standing on, and to @tonyd615 and @RandomLlama whose GB10 notes saved me time on the NCCL and cudagraph side.

2

I made a checkpoint that might load better but I haven’t had the chance to test it out on the sparks yet.

3

Thanks, that is useful. I had a look at your config and index on the hub rather than pulling the whole thing, and I think it will hit the same wall. Worth checking before you spend a download on it.

Your ignore list no longer covers the ple modules, zero entries matching, which is the sensible thing to change if you are aiming at this error. But the checkpoint still ships the same PLE layout: 128 ngram_embedding.shard_N.weight plus exactly one ngram_embedding.weight_scale, in model-00138.safetensors.

The problem is that the gate never reaches the ignore list. It decides on the config type first:

def _get_ple_embedding_quant_method(quant_config, prefix):
    if not isinstance(quant_config, Fp8Config):
        return None
    if not quant_config.is_checkpoint_fp8_serialized:
        return None
    ignored_layers = quant_config.ignored_layers
    ...
    return Qwen3_8FlashNextPLEFp8EmbeddingMethod()

Your quant_method is still modelopt, so it fails the isinstance on the first line and returns None before the ignore list is read at all. vLLM’s modelopt path does not handle embeddings either, there is no VocabParallelEmbedding anywhere in modelopt.py, so nothing else picks it up. The module is built as a plain embedding with no weight_scale slot, and loading dies on the scale that is still in your index.

Caveat that I have not run yours. This is from reading your config.json and index against that function, so I could be wrong about how it actually plays out.

If it helps I am happy to run it on the two Sparks and report back, the rig is free and the harness is set up. The only thing that would stop me is disk, since it is another 135 GB alongside the two copies I already keep local, so say the word and I will clear space rather than guess.

For what it is worth, the reshard itself may still be worth having for other reasons. My load takes 550 to 760 seconds per node from 206 shards, and fewer, larger files would likely help that.

4

Posting the FP8 side in full, since the numbers above only quote it as a comparison and the configuration is worth having on its own. Same two Sparks, same container, same harness, so the two sets are directly comparable.

Four things stopped the engine on the FP8 build. None of them says what is actually wrong.

The first is an error about a flag you never passed:

vllm serve: error: argument --compilation-config/-cc: 1 validation error
  Invalid JSON: input_value='vllm serve /model --served-model-name ...'

The image ENTRYPOINT is already vllm serve, so a -c in the docker run line is consumed as --compilation-config instead of reaching the shell. Launch with --entrypoint bash. Same fix applies to the NVFP4 build.

Second, fp8 KV is rejected:

NotImplementedError: Qwen3.8-Flash-Next QSA requires a BF16 main KV cache

Qwen Sparse Attention keeps its main KV in bf16. The stock QSA kernels declare supported_kv_cache_dtypes = [“auto”, “bfloat16”] and raise on anything else. That costs twice the KV bytes an fp8 cache would use and is the largest single memory line item on either build.

Third, the tool call parser. The Qwen repo README says qwen3_coder, the vLLM recipe says qwen3_xml, and qwen3_xml is the one that works here. This one does not raise. It produces a complete agent run in which every instance fails to parse an action, so verify it with a single request that comes back with finish_reason: tool_calls before running any evaluation. That one cost me four wasted runs.

Fourth, there is no VLLM_FLASHINFER_MOE_BACKEND environment variable in this build. The MoE backend is selectable through the --moe-backend argument instead.

A concern that did not materialise: vLLM issue 43906 describes the SM_121 MoE selector falling back to MARLIN and dequantizing FP8 experts to BF16, which on a 125B MoE inside 121 GB would be a capacity problem rather than only a speed one. It does not apply to this checkpoint:

Using DEEPGEMM Fp8 MoE backend out of potential backends:
  ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'TRITON', 'MARLIN', ...]

That issue is scoped to MXFP8, OCP block-32. This one is fine grained block-128 and unaffected.

Working configuration, identical to the NVFP4 one above apart from the model path and the served name:

--tensor-parallel-size 2 --enable-expert-parallel
--gpu-memory-utilization 0.85
--max-model-len 262144
--max-num-seqs 64
--kv-cache-dtype bfloat16
--enforce-eager
--reasoning-parser qwen3
--enable-auto-tool-choice --tool-call-parser qwen3_xml
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--distributed-executor-backend mp --nnodes 2 --node-rank 0

Weights land at 88.07 GiB per node, against 64.06 for NVFP4, and the engine reports:

GPU KV cache size: 607,890 tokens, Maximum concurrency for 262,144 tokens per request: 2.32x

Concurrency, 512 output tokens per request:

conc   TTFT mean   per-stream t/s   aggregate t/s
   1      1.320 s           24.42           23.02
   2      0.521 s           22.59           42.08
   4      1.135 s           16.70           61.90
   8      1.455 s           12.96           94.28
  16      2.459 s           10.18          145.84
  32      3.770 s            9.38          152.66
  64      3.246 s            8.46          176.75

16 is the last doubling that buys a large gain, and past it you pay per stream latency for roughly another 20 percent aggregate. I originally ran this at max-num-seqs 16 and could not tell whether the knee was the machine or the scheduler cap, so I re-ran the whole ladder pinned at 64. It lands in the same place, so it is the machine.

Context, needle in a haystack with 5 needles:

prompt tokens     TTFT    decode t/s   needles
        4,316    3.6 s         37.24   5/5
       33,371   15.5 s         37.20   5/5
      132,971   63.4 s         43.54   5/5
      202,817   95.7 s         38.15   5/5
      248,418  118.8 s         29.78   5/5

248,418 is effectively the full window with room left to generate. A 258K target fails with HTTP 400 because the rendered prompt reaches 261,583 tokens and the output no longer fits inside 262,144. Both builds behave the same way there.

On agentic quality, a 40 instance SWE-bench Pro subset scored 36 of 40 on this build and 36 of 40 on NVFP4, same harness and prompts. FP8 took 3h21m against 2h30m. They fail different instances, with 2 in common.

Short version if you are choosing between them: FP8 is the one that loads without patching vLLM. NVFP4 wins on every measurement I took and needs the one line change in the first post. Quality came out identical on the only quality benchmark I ran.

5

The reason I used the Nvidia checkpoint as the base is because they used the nemotron dataset for the calibration whereas the Radix checkpoint used only the cnn_dailymail dataset. The thinking was the Nvidia checkpoint would give better quality than the Radix checkpoint.

I haven’t had time to test out my checkpoint on the sparks yet but it runs great on a single RTX PRO 6000.