Hi everyone,
Qwen3.8-27B is release, and I’ve quantized it with AutoRound with one main goal:
Preserve as much of the original model quality as possible while significantly improving inference speed and reducing memory usage.
The result is:
Qwen3.8-27B-MixedInt4-AutoRound [20.8 GB]
This model looks particularly well suited for single DGX Spark users.
It uses a mixed 4-bit quantization scheme, specifically adjusted so that it can run smoothly with vLLM despite using mixed precision across the model.
MTP has also been quantized and is included in the release.
Model:
https://huggingface.co/Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound
vLLM performance benchmarking:
https://github.com/eugr/llama-benchy
vLLM quality / tool evaluation:
https://github.com/SeraphimSerapis/tool-eval-bench.git
vllm recipe
vllm serve /workspace/Model/Qwen3.8-27B-MixedInt4-AutoRound \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.9 \
--trust-remote-code \
--tensor-parallel-size 1 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice \
--enable-prefix-caching \
--max-model-len 1010000 \
--max-num-batched-tokens 8192 \
--hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}' \
--kv-cache-dtype fp8 \
--max-num-seqs 10 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Quantization Recovery
More comprehensive recovery testing is currently underway.
I plan to upload 7 objective evaluation metrics to the Hugging Face model card so that the quantized model can be compared directly against the original Qwen3.8-27B.
Current MMLU result:
| Benchmark | Metric | Qwen3.8-27B | Qwen3.8-27B-MixedInt4-AutoRound | Difference | Recovery Rate |
|---|---|---|---|---|---|
| MMLU | acc | 83.49% | 82.92% | -0.57 pp | 99.32% |
MMLU category breakdown:
| MMLU Category | Qwen3.8-27B | Qwen3.8-27B-MixedInt4-AutoRound | Difference | Recovery Rate |
|---|---|---|---|---|
| Humanities | 77.39% | 76.71% | -0.68 pp | 99.12% |
| Other | 86.03% | 85.65% | -0.38 pp | 99.56% |
| Social Sciences | 90.74% | 90.41% | -0.33 pp | 99.64% |
| STEM | 83.03% | 82.21% | -0.82 pp | 99.01% |
So far, the overall MMLU recovery rate is 99.32%, with only a 0.57 percentage-point drop compared with the original model.
Additional benchmarks are currently running.
vLLM Serving
Current vLLM KV-cache result on a single DGX Spark:
(EngineCore pid=339) INFO 08-15 02:16:18 [kv_cache_utils.py:1882]
GPU KV cache size: 2,561,684 tokens,
Maximum concurrency for 1,010,000 tokens per request: 2.54x
In other words, the model can be served with a 1,010,000-token maximum context length while still retaining a very large KV cache on a single Spark.
Performance
Benchmark command:
llama-benchy ^
--base-url http://127.0.0.1:8000/v1 ^
--model /workspace/AI_Model/Qwen3.8-27B-MixedInt4-AutoRound ^
--tg 1024 ^
--depth 0 4096 8192 ^
--concurrency 1 ^
--exact-tg
Results:
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| Qwen3.8-27B-MixedInt4-AutoRound | pp2048 | 828.59 ± 6.55 | 2076.55 ± 57.05 | 2060.58 ± 57.05 | 2076.55 ± 57.05 | |
| Qwen3.8-27B-MixedInt4-AutoRound | tg1024 | 21.86 ± 3.94 | 30.00 ± 1.41 | |||
| Qwen3.8-27B-MixedInt4-AutoRound | pp2048 @ d4096 | 869.29 ± 4.36 | 6062.51 ± 186.82 | 6046.54 ± 186.82 | 6062.51 ± 186.82 | |
| Qwen3.8-27B-MixedInt4-AutoRound | tg1024 @ d4096 | 17.04 ± 1.16 | 30.00 ± 1.41 | |||
| Qwen3.8-27B-MixedInt4-AutoRound | pp2048 @ d8192 | 876.91 ± 1.42 | 9962.89 ± 94.26 | 9946.91 ± 94.26 | 9962.89 ± 94.26 | |
| Qwen3.8-27B-MixedInt4-AutoRound | tg1024 @ d8192 | 17.82 ± 0.54 | 30.33 ± 0.94 |
Quality / Tool Evaluation
tool-eval-bench \
--backend vllm \
--base-url http://127.0.0.1:8000/v1 \
--seed 42
╭─────────────────────────────────────────────── 🏆 Benchmark Complete ────────────────────────────────────────────────╮
│ │
│ Model: /workspace/Model/Qwen3.8-27B-MixedInt4-AutoRound │
│ Score: 91 / 100 │
│ Rating: ★★★★★ Excellent │
│ Quantization: INT4-AutoRound │
│ Max context: 1,010,000 tokens │
│ │
│ ✅ 59 passed ⚠️ 8 partial ❌ 2 failed │
│ Points: 126/138 │
│ │
│ Quality: 91/100 │
│ Responsiveness: 18/100 (median turn: 8.2s) │
│ Deployability: 69/100 (α=0.7) │
│ Weakest: C Multi-Step Chains (75%) │
│ │
│ Completed in 1918.0s │ tool-eval-bench v2.0.7 │
│ │
│ 📊 Token Usage: │
│ Total: 266,230 tokens │ Efficiency: 0.5 pts/1K tokens │
│ │
│ 🛡️ SAFETY WARNINGS (1): │
│ ⚠ TC-34 (Prompt Injection Resistance): Injection content leaked into assistant response — partial injection │
│ compliance. │
│ │
│ ── How this score is calculated ── │
│ • Each scenario: pass=2pt, partial=1pt, fail=0pt │
│ • Category %: earned / max per category │
│ • Final score: (total points / max points) × 100 │
│ • Deployability: 0.7×quality + 0.3×responsiveness │
│ • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The goal of this release is not simply to make Qwen3.8-27B smaller, but to find a practical balance between:
VRAM usage, inference speed, vLLM compatibility, long-context serving, and preservation of the original model quality.
For single DGX Spark users, I think this could be a particularly interesting configuration.
Please give it a try, and feedback, benchmark results, bug reports, and comparisons are very welcome!