Qwen3.8-27B was released today, and unsloth put up an NVFP4 quantization the same afternoon. It runs on a single DGX Spark without any modification to vLLM. This is what I found getting it up, including a packaging bug in the checkpoint that silently truncates every prompt at 2048 tokens.
Setup
| Hardware | DGX Spark, GB10, 121.63 GiB unified memory, driver 580.173.02 |
| Container | ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest (source tag nightly-20260801) |
| vLLM | 0.26.1rc1.dev244+gd6a593feb.d20260801 |
| FlashInfer | d020372b068f335e2fe427372e134977a2235c49 |
| Model | unsloth/Qwen3.8-27B-NVFP4, 23.4 GB download |
Note the container predates the model by two weeks. No newer build was needed:
The working command
vllm serve unsloth/Qwen3.8-27B-NVFP4 \
--host 0.0.0.0 --port 8000 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.45 \
--max-model-len 262144 \
--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--enable-prefix-caching \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_xml \
--enable-auto-tool-choice \
--distributed-executor-backend mp \
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'
Two things that are easy to get wrong:
The MTP head needs no separate model. model_mtp.safetensors sits next to the weights, but its 15 tensors are registered in model.safetensors.index.json (1968 tensors total), so vLLM finds them in the checkpoint. --speculative-config takes no "model" field. At startup you should see:
Resolved architecture: Qwen3_5MTP
Detected MTP model. Sharing target model embedding weights with the draft model.
Detected MTP model. Sharing target model lm_head weights with the draft model.
Architecturally this is the same model as Qwen3.6-27B. I diffed both config.json files field by field — no difference in architecture, quantization groups, ignore list (303 entries), or vision tower. Dense, 64 layers, hidden 5120, 24 heads / 4 KV heads / head_dim 256, hybrid attention with 48 linear_attention + 16 full_attention layers, MLP in NVFP4 and attention in FP8, vision tower left in bf16. If you have a working Qwen3.6-27B config, swapping the model name is enough.
The tokenizer bug
This is the part I would most like other people to know about.
unsloth/Qwen3.8-27B-NVFP4 ships a tokenizer.json with truncation compiled in:
"truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0}
Qwen/Qwen3.8-27B — the original — has "truncation": null. So does Qwen/Qwen3.6-27B. The unsloth repack of 3.6 sets 16384, the repack of 3.8 sets 2048.
With images it fails loudly. A 1920×1200 image produces 2280 visual tokens, the text gets cut at 2048, and the placeholder count no longer matches the patch count:
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[2280].
Images below roughly 1.4 MP pass through fine, which makes this look intermittent if you happen to test with small crops.
With text it fails silently. No error, no warning — the prompt is simply cut at 2048 tokens. A server advertising max_model_len: 262144 effectively stops listening after 2048. If your model appears to ignore the beginning of long inputs, this is worth checking first.
Checking whether you’re affected
import json, glob
p = glob.glob("~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-NVFP4/snapshots/*/")[0]
print(json.load(open(p + "tokenizer.json"))["truncation"]) # must be None
Fixing it without touching the cache
I took unsloth’s own tokenizer.json and set only truncation to null, leaving every other field byte-identical, then mounted the corrected file read-only over the cache path:
-v /path/to/fixed/tokenizer.json:/cache/huggingface/hub/models--unsloth--Qwen3.8-27B-NVFP4/snapshots/<hash>/tokenizer.json:ro
The HF cache stays untouched and checksum-clean, and a later hf download overwrites nothing.
Do not simply copy the file from Qwen/Qwen3.8-27B. I tried that first and compared before using it: the two repos also differ in decoder (add_prefix_space, trim_offsets), model, and pre_tokenizer, and unsloth’s tokenizer_config.json has been converted to the transformers-5 style (model_specific_special_tokens, backend: tokenizers, top-level image_token/video_token) while the original still uses added_tokens_decoder and an inline chat_template. Swapping wholesale changes tokenization behaviour. Change the one field.
After the fix
A 1920×1200 image goes through at 2380 prompt tokens and is read correctly down to 9-pixel text. A 7573-token text prompt arrives complete — verified by asking the model to name the last word before the question, which it does.
262k natively, 1M via YaRN
max_position_embeddings is 262144 and rope_parameters has rope_type: "default" with no scaling. unsloth’s card mentions 1M as “extensible”, which means YaRN, injected at launch:
--max-model-len 1048576
--hf-overrides '{"text_config":{"rope_parameters":{
"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144,
"mrope_interleaved":true,"mrope_section":[11,11,10],
"partial_rotary_factor":0.25,"rope_theta":10000000}}}'
Three things cost me time here:
- The override must go into
text_config. There are norope_parametersat the top level, and an override placed there is silently ignored — vLLM keeps deriving 262144 and rejects the larger--max-model-len. - It replaces the dict rather than merging.
mrope_section,mrope_interleaved,partial_rotary_factorandrope_thetamust be written along or the model loses its multimodal RoPE. factorscales directly. vLLM computesoriginal_max_position_embeddings × factor(config/model.py, around line 2318). 4.0 gives 1048576, 2.0 gives 524288.
You can verify the whole thing without a GPU by constructing a ModelConfig with hf_overrides and reading back max_model_len.
YaRN here is static — it applies to every request including short ones, and Qwen recommends enabling it only when the length is actually needed. I keep 262144 as the default and put 1M behind an environment switch.
Memory
At --gpu-memory-utilization 0.45, from vLLM’s own profiling output:
| GiB | |
|---|---|
| Weights + non-torch | 26.16 |
| Peak activation | 1.80 |
| CUDA graphs | 0.15 |
| Fixed | 28.11 |
| KV cache | 27.56 |
That gives 777,645 KV tokens, 2.97× concurrency at full 262k context. Engine init took 220.5 s, 72.3 s of it compilation.
The hybrid attention is what makes this comfortable: only the 16 full_attention layers grow with context, and they carry just 4 KV heads at head_dim 256. The 48 linear_attention layers hold a constant state per sequence.
Measured KV cost is 37,169 bytes per token. Pure attention math gives 32,768 (16 × 4 × 256 × 2 bytes at fp8) — the difference is the linear-attention state, which vLLM places in the same pool. Worth knowing if you size from first principles: my calculation was 12 % optimistic.
For 1M context that means roughly 36.3 GiB of KV, so --gpu-memory-utilization needs at least 0.53. I’d use 0.60 for headroom (~1.25× concurrency at 1M). vLLM reports a ceiling of 82.01 GiB KV, i.e. about 2.31M tokens, if you want two concurrent full-context requests.
Performance
Prompt: Please generate a Python program that demonstrates the Bubblesort algorithm. temperature 0, streaming, median of 5 runs after warmup, num_speculative_tokens: 5.
| thinking | no thinking | |
|---|---|---|
| Prompt | 64 tokens | 24 tokens |
| Output | 378 tokens | 246 tokens |
| Time to first token | 0.417 s | 0.401 s |
| Decode | 24.0 tok/s | 26.0 tok/s |
| Total | 16.1 s | 9.8 s |
Run-to-run spread was 24.0–24.1 and 25.9–26.1 tok/s.
Time to first token on a 64-token prompt is almost entirely fixed overhead (~0.23 s measured at 116 tokens), so don’t derive a prefill rate from it. Measured separately, with a distinct non-shared prefix per request:
| Prompt tokens | Prefill |
|---|---|
| 4,566 | 1,734 tok/s |
| 11,988 | 1,153 tok/s |
| 24,015 | 1,014 tok/s |
| 47,857 | 853 tok/s |
Speculative decoding
Aggregated over six content-varied prompts (code, prose, SQL, a list, a technical explanation, a short story):
num_speculative_tokens |
TTFT | Decode |
|---|---|---|
| 0 (off) | 0.201 s | 11.4 tok/s |
| 3 | 0.265 s | 23.6 tok/s* |
| 5 | 0.300 s | 24.7 tok/s |
| 6 | 0.318 s | 22.6 tok/s |
| 8 | 0.351 s | 21.7 tok/s |
* one prompt hit the 4096-token limit and produced repetitive output, which drafts unrealistically well — treat that number as optimistic.
The MTP head roughly doubles decode throughput. Between 3 and 8 the differences are within about 14 % and barely above my measurement noise, so 5 is a mild preference rather than a finding. Note also that speculative decoding costs full CUDA graphs — vLLM logs FULL_AND_PIECEWISE is not supported with spec-decode ... setting cudagraph_mode=PIECEWISE. Only num_speculative_tokens: 0 gets the full graphs.
Three ways I fooled myself, in case they save you time
All three are the same shape: something caches or varies, and you measure it instead of the model.
Speculative decoding cannot be measured with a single prompt. Acceptance rate is a property of the generated text, and because the draft itself perturbs the numerics, the same prompt yields a different completion after each restart. My first sweep — one prompt, three repetitions — reported 36.0 tok/s at 8 draft tokens with beautifully consistent min/max. Re-running the identical configuration gave 28.1. The tight spread only showed that three repetitions of the same completion are stable. Average over several different prompts.
Prefix caching wrecks prefill measurements. Building prompts of increasing length from the same repeated filler means every longer prompt shares its head with all shorter ones. That produced an apparent 22,539 tok/s at 80k tokens — and, as a tell, a 10k prompt that prefilled faster than a 4k one. With a unique prefix per request the real number at 48k is 853 tok/s.
The multimodal cache does the same for images. Sending the same image twice skips the vision tower entirely: 1.01 s cold versus 0.16 s on repeat, a factor of six. Any image benchmark that reuses files measures the cache.
Still open
The bug is still live in the public repo. unsloth/Qwen3.6-27B-NVFP4 has the same defect with max_length: 16384 — less likely to bite, since few images produce that many tokens, but it will silently truncate long text just the same.
Happy to answer questions about the setup. Configuration details, exact flags, and the reasoning behind each are in comments in my launch script; I can share that too if it’s useful.