Hey everyone,
Like many of you I’ve been running Qwen3.5-122B-A10B on a single DGX Spark with the GPTQ-INT4 checkpoint. It works at ~15 tok/s, but I wanted to see if I could squeeze more out of it without going full INT4 on the attention layers.
The snap was: the GPTQ-INT4 checkpoint only quantizes the MoE experts. The attention and shared expert layers stay at BF16 — and those are the ones hit on every token. Instead of pushing them further down to INT4 (like Intel’s AutoRound approach, which has shown great results on GPQA Diamond), I wanted to try FP8 — reduce the bandwidth cost by half while keeping closer to the original precision. The tradeoff is intentional: prioritize precision on the layers where it matters most (attention) over squeezing out every last tok/s.
The result is 21.5 tok/s (46.4ms ITL P50), up from 15. Not as fast as uniform INT4 would be, but it preserves the official calibrated FP8 precision on the dense layers. Sharing the approach here in case it’s useful.
The bandwidth picture
The GPTQ-INT4 checkpoint only quantizes the MoE expert weights to INT4. The dense layers — attention projections, shared experts — stay at BF16 (2 bytes/param). On a per-token basis:
- MoE experts (INT4): ~2.4 GB read per token (only 8 of 128 experts are active)
- Dense layers (BF16): ~18.2 GB read per token (attention + shared experts, every token)
- Total: ~20.6 GB/token
With Spark’s 273 GB/s memory bandwidth: 273 / 20.6 ≈ 13 tok/s theoretical, matching the measured 15 tok/s.
The experts aren’t the bottleneck — the BF16 dense layers are. INT4 experts are already very compact (0.5 bytes/param). But the dense layers at 2 bytes/param account for ~88% of per-token bandwidth.
The fix: FP8 for dense layers, INT4 for experts
The official Qwen/Qwen3.5-122B-A10B-FP8 checkpoint has calibrated FP8 block-128 weights with proper weight_scale_inv tensors — but at 127 GB it won’t fit on a single Spark at useful context lengths.
The idea: take the GPTQ-INT4 checkpoint’s expert weights (already compact) and replace the BF16 dense layers with FP8 versions from the official FP8 checkpoint. This cuts the dense layer bandwidth from 18.2 GB to ~9.1 GB per token:
- MoE experts (INT4): ~2.4 GB/token (unchanged)
- Dense linears — attention Q/K/V/O, shared expert gate/up/down (FP8): ~9.1 GB/token (half of BF16)
- Norms, routing gates, embeddings, DeltaNet small projections (BF16): ~0.5 GB/token (too small to matter)
- Total: ~12 GB/token → 273 / 12 ≈ 23 tok/s theoretical
Not everything becomes FP8. The 307 large linear layers (attention projections and shared experts) get FP8 — they’re the bandwidth hogs. The small stuff (RMSNorm weights, MoE routing gates, DeltaNet conv1d/in_proj, embeddings, lm_head) stays BF16 because they contribute < 3% of per-token bandwidth and quantizing them would risk quality for negligible gain.
Step 1: Build the hybrid checkpoint
I wrote a script that:
- Downloads only the 3 shards (of 64) from
Qwen/Qwen3.5-122B-A10B-FP8that contain non-expert tensors - Extracts 307 FP8 weight + scale tensor pairs (attention projections, shared expert linears)
- Replaces the corresponding BF16 tensors in the GPTQ-INT4 checkpoint
- Rebuilds the safetensors index
Result: 70 GB checkpoint (down from 74 GB) with INT4 MoE experts + FP8 dense layers + BF16 norms/gates.
Step 2: Make vLLM use FP8 GEMMs for those layers
This is where it gets interesting. vLLM picks one quantization method per model. When you load with --quantization gptq_marlin, layers excluded from GPTQ fall back to UnquantizedLinearMethod, which casts FP8 tensors to BF16 at load time. You save VRAM but get zero bandwidth improvement — the matmuls still run at BF16.
I patched two files in vLLM (v0.17.1, ~150 lines total) to make GPTQMarlinConfig auto-detect FP8 layers and route them to Fp8LinearMethod instead:
gptq_marlin.py — maybe_update_config() scans safetensors metadata for float8_e4m3fn weights paired with weight_scale_inv tensors, infers the block size from shape ratios, and constructs an Fp8Config. Layer names get mapped through the model’s WeightsMapper so it works with any model architecture (not hardcoded to Qwen3.5).
gptq_utils.py — get_linear_quant_method() checks for FP8 layers before falling back to UnquantizedLinearMethod. Handles fused modules (qkv_proj, gate_up_proj) correctly via packed_modules_mapping.
No new config names, no new CLI flags. Just --quantization gptq_marlin as usual — the FP8 detection is automatic.
At inference time, each layer hits a different compute path:
| Layer type | Examples | Quant method | GEMM kernel | Bytes/param |
|---|---|---|---|---|
| MoE experts | mlp.experts.*.gate_proj |
GPTQMarlinMoEMethod | Marlin INT4 | 0.5 |
| Dense linears (307 layers) | self_attn.q_proj, shared_expert.down_proj |
Fp8LinearMethod | CUTLASS W8A8 block FP8 | 1.0 |
| Small weights | RMSNorm, routing gates, embeddings, lm_head | UnquantizedLinearMethod | BF16 GEMM | 2.0 |
The BF16 layers are still there — we didn’t eliminate them. We just made them a rounding error (< 3% of per-token bandwidth) instead of the dominant term (88% before).
Results
| Metric | GPTQ-INT4 (before) | Hybrid GPTQ+FP8 (after) | Change |
|---|---|---|---|
| Decode | 15.0 tok/s | 21.5 tok/s | +43% |
| ITL P50 | 67 ms | 46.4 ms | -31% |
| ITL P99 | 67 ms | 48.4 ms | -28% |
| Model memory | 68.4 GiB | 64.1 GiB | -4.3 GiB |
| KV cache | 25.8 GiB | 31.4 GiB | +21% |
All measurements with CUDA graphs, 256k max context, kv-cache-dtype fp8, single DGX Spark. Consistent 21.5 tok/s across 256, 512, and 1024 token generations. FP8 weights sourced from the official calibrated Qwen/Qwen3.5-122B-A10B-FP8 checkpoint — quality benchmarks (e.g. GPQA Diamond) not yet run.
Reproduction
Build the hybrid checkpoint:
The builder script downloads only the needed shards from the FP8 checkpoint (~8 GB), extracts the non-expert FP8 tensors, and grafts them into your existing GPTQ-INT4 checkpoint:
python3 build-hybrid-checkpoint.py \
--gptq-dir ~/models/Qwen3.5-122B-A10B-GPTQ-Int4 \
--fp8-repo Qwen/Qwen3.5-122B-A10B-FP8 \
--output ~/models/qwen3.5-122b-a10b-fp8hybrid
Install the patched vLLM fork:
The vLLM changes live in a fork. It’s purely additive — no changes to existing GPTQ behavior for models without FP8 tensors. Comes with 19 unit tests.
cd vllm
git checkout v0.17.1-hybrid-fp8
pip install -e .
Launch:
vllm serve ~/models/qwen3.5-122b-a10b-fp8hybrid \
--served-model-name Qwen/Qwen3.5-122B-A10B \
--quantization gptq_marlin \
--gpu-memory-utilization 0.85 \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--max-num-seqs 1
You should see in the logs:
Hybrid GPTQ+FP8: detected 307 FP8 layers (block_size=[128, 128]). These will use Fp8LinearMethod.
Repo: github.com/rmstxrx/vllm-hybrid-quant — checkpoint builder and documentation. The vLLM changes (2 files patched, 19 unit tests) live in the fork. Planning to submit this as a vLLM upstream PR as well — the approach generalizes to any MoE model where dense layers bottleneck decode.
Happy to answer questions or help anyone reproduce this.
Hardware: DGX Spark GB10, 128 GB unified LPDDR5X, 273 GB/s
Software: vLLM v0.17.1 (patched fork), CUDA 13.0, FlashInfer attention backend
Model: Qwen3.5-122B-A10B (122B total / 10B active per token)
Context: 256k max, FP8 KV cache
Edit 2026-03-18 14:33 UTC -4: Repo restructured after code review. The vLLM patch now lives in a proper fork (rmstxrx/vllm branch v0.17.1-hybrid-fp8) instead of a standalone .patch file — easier to clone and install. Bug fixes applied: FP8 detection now works for all GPTQ checkpoint variants (not just those without pre-set module lists), layer matching uses exact name comparison instead of substring, block-size validation enforces uniformity across all FP8 layers. Test count increased from 16 to 19. Checkpoint builder hardened with input validation, shape checks, and output directory safety. The standalone repo (vllm-hybrid-quant) now contains only the checkpoint builder + docs. Updated line count (~100 → ~150), test count, and reproduction instructions above to match.
