I know MTP is mathematically lossless, yet in all the benches I did, I observed a quality change , not necessarily negative that could not be explain. I’ve tried various options, including temp, etc but it could be as much as a 5 points differences in tool call bench which cannot be explained by noise or SD.
Any thoughts ?
MTP being ‘mathematically lossles’ is categorically wrong and is an urban legend.
The drafter generates some outputs and the main model acceps or rejects them. This is not a lossless process, since the main model will be biased when it sees the drafted generation, as opposed to generating itself. This is usually negative, because weaker drafters are more likely to propose a fast but sub-optimal next tokens, compared to the main model guessing the token itself. The main model might accept this next token even if it is sub-optimal, statistically making MTP enabled versions worse than those without.
Also, tool-call-eval is a great bench, but doesn’t tell you A LOT about the “intelligence” or capabilities of the model. I’ve ran benchmarks with and without MTP in my custom suite, and I’m able to pinpoint minor differences, mostly regarding to run variance though. But still, the same model with and without MTP end up scoring ALMOST identical in capability testing.
To me, it’s a tradeoff worth having (i.e.: 40% improvement on speed vs a ~2% hit in quality for Qwen3.6-27B in my tests)
I spent great time improving mtp by adjusting prefix-batch size,. It depends on model (attention type, heads, attention cache size) and number of prediction tokens. For DS4F optimal size is 16384 prefix-batch and MTP 4 for 70-75% stable prediction quality, goes up to 80% on coding, goes down to 70% on llama-benchy performance tests.
I don’t have strong evidence how it affects quality, but the hunch is it does, it should not but it depends on quality of serving stack, how attemtion threats discarded block, how it re-evaluates. Many holes to fall through. I just elected to focus on prediction improvement and balance it with cache size, as prefix-batch size greatly eats into kv cache.
If MTP is properly implemented in the inference engine’s serving stack, the target model isn’t biased at all by the drafted outputs. Causality between the drafted tokens and the target verify step of the base model is strictly enforced. MTP is in fact mathematically lossless. Even if the drafter model is entirely misaligned to the target model with 0% acceptance rates, you would only observe a throughput degradation, not a quality degradation. If the drafter model emits a sub-optimal token which the target model accepts, the probability that the target model accepts said token would have been the identical probability that it output the same token in absence of the MTP drafter.
I think the problem right now is that both vllm and llama.cpp have bugs relating to MTP and prefix caching. So if you’re not using prefix caching then you’re probably not seeing any noticeable degradation. Otherwise I typically leave MTP off for my normal agentic workflows.
I love MTP as much as the next person for the speed it brings to the table, use it in deployment, but also we must acknowledge the elephant in the room (that in my opinion not enough people addressing), theory != deployment.
You are absolutely correct about the math of theoretical speculative decoding. But you are ignoring the architecture of the models we actually deploy. In production, you can have strict mathematical losslessness, or you can have MTP-driven speedups. Because of the early specialization of NTP layers, you cannot currently have both. If you enforce strict mathematical verification, the inherent decay of the MTP heads guarantees terrible acceptance rates and throughput degradation. To avoid that degradation and keep the servers fast, any real-world deployment is lossy by design.
I am coming across research papers since early 2025 like this one:
And here is one of the most recent ones:
I would be happy to be proven wrong, have my cake and eat it too. But as far as I can see, any and all practical MTP implementations are inherently corner cutting exercises.
Working with LLM, we would first have to accept that it’s probabilistic rather than deterministic but to suggest “MTP being ‘mathematically lossles’ is categorically wrong and is an urban legend.” probably would insult a great many people smartner than me.
You are suggesting that in a coin flip where the probability is 50% , if drafter consistently generates “Head” for whatever reason , that the model will accept always accept “head”. I’m sure that there are “lossy implementation” but that’s not how I understand speculative decoding works and as per doc, not how vLLM is implemented.
Sorry for opening the can of worms, “theory != deployment” is too general a statement but that’s like saying that the head isn’t the body, right yet so wrong. Deployment should always be based on solid understanding of the theory, or at least on the creation of somebody else with a solid understanding of the theory. There are “real world” auxillary aspect of deployments, things that I might not have consider, but there are always further theory of how these auxillary components work, just because I don’t have a fuller understanding of these components, doesn’t means that there isn’t a functional theory behind how they operate.
The bugs seems to be in scenerio where both MTP and prefix caching are used together. I mostly leave prefix caching off, but it’s something worth further verifications on different workload.