Goal: Abliterated Llama-4-Scout-17B-16E-Instruct → NVFP4 (W4A4, compressed-tensors) to serve on SGLang v0.5.12-cu130 on a GB10 / sm_121 (flashinfer_cudnn FP4 backend).
Pipeline: p-e-w/heretic abliteration → 13.6 MB LoRA adapter (rank 3; targets self_attn.o_proj on all 48 layers + feed_forward.shared_expert.down_proj). Then merge adapter into base → llm-compressor NVFP4, using RedHat’s published Scout recipe (QuantizationModifier(scheme="NVFP4"), ignore lm_head/self_attn/router/vision_model/multi_modal_projector, sequential_targets=["Llama4TextMLP"], custom SequentialLlama4TextMoe that un-fuses experts and frees the fused tensors, neuralmagic/calibration dataset).
Symptom: Model loads cleanly on SGLang (CompressedTensorsW4A4Nvfp4MoE, “fired up”), but every prompt returns pure � (U+FFFD) gibberish.
Ruled out:
-
SGLang config — RedHat’s vanilla
Llama-4-Scout-17B-16E-Instruct-NVFP4works perfectly on the identical SGLang config/node. -
Format — our checkpoint’s
model.safetensors.index.jsontotal_size= 69.7 GB / 14 shards, parameter dtype counts identical to RedHat (U8 51.34B, F8_E4M3 6.42B, BF16 5.97B),weight_scaleper-block tensors present. -
MoE forward — our custom
SequentialLlama4TextMoe.forwardverified mathematically identical to transformers 4.56.2’sLlama4TextMoe.forward.
Version findings: llmcompressor 0.10.0.2 (ct 0.14.0.1) and 0.9.0 (ct 0.13.0) drop the per-block weight_scale → gibberish. 0.8.1 (ct 0.12.2) writes weight_scale and produces the correct 69.7 GB — but still gibberish.
The one remaining difference from RedHat: ours is the abliteration-merged model quantized; RedHat’s is vanilla. Adapter itself is validated (generated coherently at 86->16 refusals during abliteration).
Questions: (1) Is merge-then-NVFP4 on Llama4 known to break this way?
(2) Should the LoRA be merged under a specific transformers version before quant?
(3) Is there a known-good llmcompressor version for Llama4 NVFP4 that keeps weight_scale and correct packing?
(4) Any gotcha quantizing a merged/abliterated Llama4 MoE vs vanilla?