I wanted Qwen3.8-27B to keep its full native 262,144-token context, embedded MTP, and optional vision while the target model and KV cache stayed on one 24 GB RTX PRO 4000 Blackwell SFF. The ready-made NVFP4 GGUF looked like the obvious answer.
It wasn’t.
On the same short WikiText-2 control, that build scored 6.4949 PPL. Plain Q4_0 scored 6.3798 and an iMatrix IQ4_XS reference scored 6.1175. The FP4 kernels were fast, but requantizing sensitive attention and Gated DeltaNet tensors to NVFP4 cost too much quality.
So I built a mixed 5.01 BPW GGUF. Large tolerant matrices remain native NVFP4. The iMatrix ranking moves selected attention, DeltaNet, and late FFN tensors to Q5_K or Q6_K; embeddings use Q6_K and the output head stays Q8_0. The trained one-layer MTP head is embedded in the same file.
The resulting model is 17,125,207,136 bytes. Its PPL was 6.1197 versus 6.1127 for Q4_1, a 0.11% difference on this control. It also left enough VRAM for the context, recurrent state, CUDA graphs, and server overhead.
Hardware and runtime
- GPU0: RTX PRO 4000 Blackwell SFF, 24,467 MiB reported, sm120a. Target, MTP, recurrent state, graphs, and KV cache live here.
- GPU1: RTX 2000 Ada. It holds the optional F16 vision projector; text-only serving does not need it.
- Debian 13, CUDA 12.9.86, GCC 14.2.
- llama.cpp with pinned experimental patches for Gated DeltaNet, speculative verification, chained MTP, recurrent checkpoint correctness, and Flash Attention swizzling.
- Q4_0 target KV, F16 draft KV, four recurrent checkpoints, batch 512, ubatch 256.
Results
| Gate | Result |
|---|---|
| Production profile, 10 runs | 50.441 tok/s mean, 49.420 to 51.397 |
| Clean llama.cpp master | 45.422 tok/s |
| Selected custom runtime | 55.402 tok/s, +21.97% |
| Target-only greedy | 21.189 tok/s |
| Embedded MTP | 59.456 tok/s, 2.81x |
| Full-context prefill | 226.750 tok/s |
| Decode after filling 261,500 tokens | 12.606 tok/s |
| GPU0 after the full fill | 23,952 / 24,467 MiB |
These are separate gates. I am not multiplying unrelated best cases into one headline number.
The 256K test was a real fill, not a successful allocation followed by a short prompt. I submitted 261,500 input tokens, generated 256 more, checked for truncation, then reused the hot cache. It completed without OOM and left about 515 MiB of physical margin on GPU0.
MTP had an unexpected sweet spot
The first sweep favored n_max=3. Values 4 through 7 slowed down as rejected draft work accumulated. Then n_max=8 jumped from 29.10 tok/s at n=7 to 49.31 tok/s. N=9 was no faster, while every extra candidate added roughly 150 MiB. N=10 crossed a CUDA allocation cliff in the full 256K profile.
More MTP precision made things worse. With all non-MTP tensors held byte-identical, replacing the NVFP4 MTP weights with iMatrix Q5_K added 50.6 MiB and reduced throughput by 3.39%. A Q5_K/Q6_K variant added 69.2 MiB and fell from 50.44 to 37.02 tok/s. Its acceptance dropped from 48.33% to 33.07%.
My working explanation is quant-drafter alignment: the embedded MTP head only needs to predict this quantized target. A drafter that is individually closer to BF16 can still agree less often with an NVFP4-heavy target.
There is one result I do not want to hide. Target-only greedy and MTP n=8 do not produce bitwise-identical continuations on the quantized target, consistent with llama.cpp batch-invariance issue #25618. Each mode is internally deterministic in my tests, but n=8 should not be described as distribution-preserving. N=1 is the conservative setting when that matters.
I released the GGUF, optional F16 projector, SHA-256 sums, tensor-type map, and patch manifest here:
The complete write-up, including the losing builds and exact server arguments:
This is a community GGUF, not an official NVIDIA Qwen3.8 checkpoint.
I am especially curious whether the n=8 jump reproduces on sm121/RTX 5090. If someone has tried a different MTP quant against this exact target, or gets access to an official NVIDIA NVFP4 checkpoint, I would like to compare acceptance and quality under a genuinely occupied long context.