This is a fascinating new approach which basically does drafting for speculative decoding on a totally separate GPU.
Right now it’s hot off the presses and fairly limited unless you like a particular 70b Llama variant and have several H100s lying around. But conceptually, I think this could leverage something like a 3090 or 4090 to draft at hyper speed with a smaller model while the Spark holds the big weights.
One to watch. I actually have a 3090 in a host with a 100GbE Mellanox card and once it’s a little more mature, probably will play with it.
Had not seen this, exciting, thank you! Their 27B repo is gated, I’ve requested access. It sounds like they’re planning a 122B option also.
I think it should “just work” with vLLM once my HF access is approved. No idea what the ideal draft length would be, but based on the comment thread showing FP8 draft performance is basically identical to BF16, my plan is to use it with the FP8 model.
I have used Qwen’s standard MTP with FP8 and the acceptance rate for dflash’s position 6-7 is about the same as seen for standard MTP position 2. No idea what will be optimal overall, though. With MTP on Spark, Qwen3.5-27B-FP8 is about 11-12 tok/s.