Update for NVFP4 Model conversion to use W4A16 instead of W4A4

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Update worth sharing back here: the four NVFP4 quants from these
threads (Anubis-Pro-105B, Behemoth-X-123B, plus DeepSeek-R1-Distill-
Llama-70B and Llama-3.3-70B-Instruct that landed since) are now
NVFP4A16 (W4A16) on Hugging Face instead of W4A4.

Triggered by a Discord conversation with @PhaelonBeaver (who wrote
the KLD PR in vLLM) and @mratsim — they pointed out two things I
hadn’t thought through carefully:

  1. NVFP4 weight storage is identical between W4A4 and W4A16
    serving modes. The “calibration” that modelopt does only sets
    activation scales; the weight bytes don’t change. Switching
    between modes is a config.json patch, not a re-quantization.

  2. KLD scores show W4A4 is 2-4× worse than W4A16, especially
    past ~10K context where activation quantization noise
    compounds with KV-cache lookups and the dense path falls
    apart for nuanced contexts.

The config patch is small — drop input_activations to null in the
quantization_config block; vLLM’s compressed-tensors loader (with
quant_method: modelopt) then takes the W4A16 path. No re-download
required for existing users; only config.json changed.

Re-benched all four on a single DGX Spark (vLLM 0.20.2rc1, MARLIN

  • FlashInfer env vars). 5-run median for short context (200 tok out
    on short prompt), single run for ~2.2K prefill → 200 out:

    Model W4A4 (was) W4A16 (now) Long ctx
    Anubis-Pro-105B-NVFP4 3.78 tok/s 3.84 tok/s (+1.6%) 3.28 tok/s
    Behemoth-X-123B-NVFP4 3.21 tok/s 3.25 tok/s (+1.2%) 2.74 tok/s
    DeepSeek-R1-Distill-70B-NVFP4 (no prior) 5.75 tok/s 4.88 tok/s
    Llama-3.3-70B-Instruct-NVFP4 (no prior) 5.69 tok/s 4.75 tok/s

Decode speed essentially unchanged (within run-to-run variance which
itself was <1% for all four — median/min/max ranges identical to the
second decimal). The W4A16 path was actually marginally faster on
both models where we have a direct comparison. The expected quality
gain is the headline; speed didn’t pay a price.

Cold load (full disk-to-serve including MARLIN kernel JIT):
DeepSeek-R1-Distill-70B: 361s (40 GB shards)
Llama-3.3-70B-Instruct: 400s (40 GB shards)
Anubis-Pro-105B: 542s (58 GB shards)
Behemoth-X-123B: 435s (66 GB shards)

Quality side hasn’t been A/B’d against Q6_K imatrix yet — that’s
the proper test and I’d genuinely value seeing it from anyone in
the community who runs RP or coding workloads regularly.

The pipeline-side takeaway for new quants going forward: emit
NVFP4A16 from the start unless there’s a specific reason to want
the dynamic-input-quant path (mostly compute-bound short-prompt
serving where the ~10-20% decode speed difference matters more
than the quality gap). Anubis-Pro and Behemoth model cards have
been updated; pipeline (GitHub - KaletoAI/distrib-nvfp4: Distributed NVFP4 quantization pipeline for 100B+ LLMs on a 2-node NVIDIA DGX Spark cluster · GitHub) will
get a flag in the next iteration so users don’t even have to
think about it.

For anyone with existing modelopt-exported NVFP4 quants on disk,
the minimal patch is:

  • quant_method: “modelopt” (no change)
  • config_groups.group_0.input_activations: null (was {num_bits:4, …})

That’s it. vLLM loader handles the rest correctly.

Kai

5 Likes