For everyone looking for the dflash version of qwen3.6-27b without waiting the 3 years for Mitko Vasilev to upload his stuff go to here: z-lab/Qwen3.6-27B-DFlash · Hugging Face.
They seem to have forgotten to mark it as a dflash model in terms of their categories so it is basically unfindable in google, enjoy! I will have dflash + ddtree implemented by tomorrow hopefully, claude did most of the implementation but there’s some things it just doesn’t understand.
Nice! DDTree… in vLLM I hope? That’s what we really need.
Yup in vLLM, I have it mostly working at this point but my bed is calling me lol
I ran a few benchmarks with this yesterday to compare to the others. The gains for bfcl were decent (and got exactly the same score), but on AgentBench there was no increase. I’m currently only running each sample 3 times (and taking the mean score) so it might just be randomness, but once I’ve found a few more benchmarks that work reliably I might start increasing the number of runs to ensure the numbers are fairly stable (and if it comes to it, I’ll start reviewing logs, but I’m really hoping the numbers will be reliable enough to avoid this 😄).
| name | AgentBench | bfcl |
|---|---|---|
| Qwen3.6 27B | 59.3% 2h 41m |
77.3% 1h 13m |
| Qwen3.6 27B speculative-config=dflash(15) |
58.0% 2h 42m |
77.3% 48m 58s |
| Qwen3.6 27B FP8 | 58.7% 1h 44m |
75.3% 37m 26s |
| Qwen3.6 27B enable_thinking=False |
56.0% 1h 40m |
78.0% 11m 52s |
| Qwen3.6 35B-A3B FP8 | 55.3% 2h 9m |
78.0% 17m 3s |
| Qwen3.6 35B-A3B | 52.7% 2h 34m |
78.0% 25m 5s |
| Qwen3.6 35B-A3B NVFP4 | 52.7% 2h 0m |
77.3% 18m 32s |
| Gemma4 31B | 45.3% 2h 4m |
77.3% 19m 49s |
| Qwen3 Coder Next FP8 | 46.0% 32m 49s |
|
| Gemma4 26B-A4B | 44.0% 2h 16m |
I think the dflash model was not trained on a very big dataset and that’s why you’re seeing bias there with the speed being better on bfcl but not on AgentBench. I have been really struggling to see any improvement with the ddtree implementation because the token acceptance drops off a sharp cliff at 6-7. I’ll still package everything up and throw it on github later so people can mess around with it with other models (35B is insanely fast with this) but until we have a stronger dflash model for 27B I don’t think it’s really going to do much for us.
Ah, that makes sense - and I realise now the HF page does say “This model is still under training” so maybe it’ll get better. I’ll try the MoE version out at some point (which doesn’t have this disclaimer) to see if it’s more general there.
Yeah, I was expecting a little better performance out of Dflash, so hopefully they drop a more thoroughly trained model.
new z-lab/Qwen3.6-27B-DFlash weights have been just been updated on HF, acceptance rates look to have improved.
I will download again, my acceptance is around 7%.. quite useless at the moment. they said they are working in some vllm sliding window attention feature as well. (hf question)
Thanks - I will re-run the benchmarks on this to see how things change :)
Try
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-
DFlash", "num_speculative_tokens": 5}'
dual spark, Qwen/Qwen3.6-27B-FP8 dflash=5:
── Run 1/2 ──────────────────────────────────────
[Q&A] 256 tokens in 6.23s = 41.0 tok/s (prompt: 23)
[Code] 512 tokens in 11.43s = 44.7 tok/s (prompt: 30)
[JSON] 1024 tokens in 21.11s = 48.5 tok/s (prompt: 48)
[Math] 64 tokens in 1.37s = 46.7 tok/s (prompt: 29)
[LongCode] 2048 tokens in 39.26s = 52.1 tok/s (prompt: 37)
── Run 2/2 ──────────────────────────────────────
[Q&A] 256 tokens in 5.94s = 43.0 tok/s (prompt: 23)
[Code] 512 tokens in 12.10s = 42.3 tok/s (prompt: 30)
[JSON] 1024 tokens in 20.26s = 50.5 tok/s (prompt: 48)
[Math] 64 tokens in 1.35s = 47.4 tok/s (prompt: 29)
[LongCode] 2048 tokens in 39.91s = 51.3 tok/s (prompt: 37)
Qwen/Qwen3.6-27B-FP8 MTP=3
── Run 1/2 ──────────────────────────────────────
[Q&A] 256 tokens in 8.02s = 31.9 tok/s (prompt: 23)
[Code] 512 tokens in 16.57s = 30.8 tok/s (prompt: 30)
[JSON] 1024 tokens in 28.11s = 36.4 tok/s (prompt: 48)
[Math] 64 tokens in 1.76s = 36.3 tok/s (prompt: 29)
[LongCode] 2048 tokens in 60.43s = 33.8 tok/s (prompt: 37)
── Run 2/2 ──────────────────────────────────────
[Q&A] 256 tokens in 7.86s = 32.5 tok/s (prompt: 23)
[Code] 512 tokens in 15.74s = 32.5 tok/s (prompt: 30)
[JSON] 1024 tokens in 27.70s = 36.9 tok/s (prompt: 48)
[Math] 64 tokens in 1.87s = 34.2 tok/s (prompt: 29)
[LongCode] 2048 tokens in 60.30s = 33.9 tok/s (prompt: 37)
upd:
or dflash=6
── Run 1/2 ──────────────────────────────────────
[Q&A] 256 tokens in 5.65s = 45.3 tok/s (prompt: 23)
[Code] 512 tokens in 10.96s = 46.7 tok/s (prompt: 30)
[JSON] 1024 tokens in 19.61s = 52.2 tok/s (prompt: 48)
[Math] 64 tokens in 1.18s = 54.2 tok/s (prompt: 29)
[LongCode] 2048 tokens in 39.90s = 51.3 tok/s (prompt: 37)
── Run 2/2 ──────────────────────────────────────
[Q&A] 256 tokens in 5.93s = 43.1 tok/s (prompt: 23)
[Code] 512 tokens in 12.99s = 39.4 tok/s (prompt: 30)
[JSON] 1024 tokens in 17.60s = 58.1 tok/s (prompt: 48)
[Math] 64 tokens in 1.26s = 50.7 tok/s (prompt: 29)
[LongCode] 2048 tokens in 42.04s = 48.7 tok/s (prompt: 37)
I had terrible issues with spec decoding on both the moe and dense Qwen3.6’s - quality dropped significantly - I mindlessly used --speculative-config ‘{{“method”:“qwen3_next_mtp”,“num_speculative_tokens”:2}}’
I saw poor coding, but also a measurable drop in things that require more creativity, like architectural design (I’ll publish those findings shortly).
Apart from quality I found that using Qwen’s recommended sampling params for Qwen35-27B, it would get caught in loops - to fix that I put min_p: 0.08 - which is not supported by spec decoding.
I understand Qwen uses NGram and it’s just not suitable for coding - but - I know very little about this - was, as you pointed out above, struggling to find information via google.
The big question then - is dflash good for surgical, precise coding and for other more creative but still very precise work like technical documentation, design etc.?
My understanding is that speculative decoding is lossless. The predicted tokens are checked against the main models output (but multiple predicted tokens can be checked in one go), and any incorrectly predicted tokens are never used.
This video explains it well:
MTP doesn’t use a separate model, but it should still be lossless. Unless there are bugs, I think your difference might be caused by something else.
Yep, my benchmark scores are roughly the same with speculative decoding or without. There is always some variance between runs, sometimes dflash does better sometimes worse by a few points, but overall its always in the same ballpark.
I tried pulling the latest version here, but it seems to keep eating up all my RAM (I’m running with the max set to 0.85, but it keeps getting up to around 99% and getting 0 scores, so I’ve been terminating it to try again). I’m not sure what’s going on here, but I might abort testing this one for now and come back to it when the disclaimer on the model page is gone 🙃
If it helps, these are my recipes (for eugr’s vllm docker builds) which seem to be working well:
27B BF16:
name: Qwen/Qwen3.6-27B
recipe_version: “1”
description: “vLLM serving Qwen/Qwen3.6-27B with MTP speculative decoding, 262K context, tool calling”
model: Qwen/Qwen3.6-27B
container: vllm-node-tf5
build_args:
–tf5
defaults:
port: 8000
host: 0.0.0.0
gpu_memory_utilization: 0.7
max_model_len: 262144
max_num_batched_tokens: 32768
max_num_seqs: 4
tensor_parallel: 2
env:
VLLM_MARLIN_USE_ATOMIC_ADD: 1
VLLM_USE_V1: 1
HF_TOKEN: <insert your HF key here>
command: |
vllm serve Qwen/Qwen3.6-27B
-O3
–max-model-len {max_model_len}
–max-num-seqs {max_num_seqs}
–enable-prefix-caching
-tp {tensor_parallel}
–gpu-memory-utilization {gpu_memory_utilization}
–port {port}
–host {host}
–load-format fastsafetensors
–enable-chunked-prefill
–enable-auto-tool-choice
–distributed-executor-backend ray
–tool-call-parser qwen3_coder
–reasoning-parser qwen3
–max-num-batched-tokens {max_num_batched_tokens}
–trust-remote-code
–default-chat-template-kwargs ‘{{“preserve_thinking”: true}}’
–attention-backend flash_attn
–speculative-config ‘{{“method”: “dflash”, “model”: “z-lab/Qwen3.6-27B-DFlash”, “num_speculative_tokens”: 10}}’
–generation-config auto
–override-generation-config ‘{{“temperature”: 0.6, “top_p”: 0.95, “top_k”: 20, “min_p”: 0.0, “presence_penalty”: 0.0, “repetition_penalty”: 1.0}}’
recipe_version: ‘1’
name: Qwen/Qwen3.6-27B
cluster_only: false
solo_only: false
35B A3B BF16:
Recipe: Qwen/Qwen3.6-35B-A3B-FP8 (DGX Spark Optimized)
Optimized for NVIDIA Blackwell (GB10/GB200) architectures
Uses native FP8 quantization and FP8 KV cache for maximum throughput.
Leverages mods/gpu-mem-util-gb for stable unified memory management.
recipe_version: “1”
name: Qwen36-35B-A3B-BF16-Dflash
description: Optimized vLLM serving for Qwen3.6-35B-A3B-BF16 on Blackwell DGX Spark
HuggingFace model to download
model: Qwen/Qwen3.6-35B-A3B
Container image to use
container: vllm-node-tf5
Mods: gpu-mem-util-gb for GiB-based reservation, fix-qwen3.5-chat-template for prompt stability
Default settings
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 2
gpu_memory_utilization: 0.7
max_model_len: 262144
max_num_batched_tokens: 16384
Environment variables
env:
VLLM_USE_V1: 1
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: 1
HF_TOKEN: <HF token>
The vLLM serve command template
command: |
vllm serve Qwen/Qwen3.6-35B-A3B
-O3
–served-model-name Qwen3.6-35B-A3B-BF16-Dflash
–host {host}
–port {port}
–max-model-len {max_model_len}
–max-num-batched-tokens {max_num_batched_tokens}
–gpu-memory-utilization {gpu_memory_utilization}
–load-format fastsafetensors
–enable-chunked-prefill
–enable-prefix-caching
-tp {tensor_parallel}
–enable-auto-tool-choice
–reasoning-parser qwen3
–tool-call-parser qwen3_coder
–override-generation-config ‘{{“temperature”: 0.6, “top_p”: 0.95, “top_k”: 20, “min_p”: 0.0, “presence_penalty”: 0.0, “repetition_penalty”: 1.0}}’
–default-chat-template-kwargs ‘{{“preserve_thinking”: true}}’
–speculative-config ‘{{“method”: “dflash”, “model”: “z-lab/Qwen3.6-35B-A3B-DFlash”, “num_speculative_tokens”: 10}}’
–attention-backend flash_attn
–generation-config auto
–override-generation-config ‘{{“temperature”: 0.6, “top_p”: 0.95, “top_k”: 20, “min_p”: 0.0, “presence_penalty”: 0.0, “repetition_penalty”: 1.0}}’
–distributed-executor-backend ray
recipe_version: ‘1’
name: Qwen3.6-35B-A3B-BF16-Dflash
cluster_only: false
solo_only: false
There’s an important caveat to this which is some speculative decoding frameworks require greedy decoding mode. When sampling, the speculative heads otherwise don’t know what decision the model will make. Among other things, disabling the sampling obviates most other sampling parameters including presence_penalty, repetition_penalty, etc. which Qwen3.x really need. And the inference engine may nearly silently force greedy mode.
In such cases, greedy may be set by default and that - rather than the speculation - is what causes looping. Thus, the output is technically lossless but behaves different in practice. The drafter is not at fault but may be presumed to be.
You may want to edit the second recipe and remove your HF token