… after Intel “handed” NVIDIA a rather convenient INT4 weight quantization story via the
Marlin kernel lineage (Autoround), Google has now returned the favour on the KV cache side —
with TurboQuant, published March 24 (ICLR 2026).
The short version: 3.5-bit KV cache, provably lossless, no calibration, no codebook,
online per-token. At 2.5 bits you start to feel it. At 3.5 bits, benchmarks on
LongBench / Needle-in-a-Haystack / RULER are statistically indistinguishable from BF16.
Not “within 1%” — statistically identical (benchmarked).
For those of us running inference on unified LPDDR5X where every GB/s counts twice
(once for capacity, once for bandwidth), this is not a minor update. A 128k-context
32B session drops from ~30 GB KV cache to ~6 GB. The rest is arithmetic.
The algorithm (PolarQuant + QJL residual correction) has no exotic hardware
dependencies — no Tensor Core calls, no arch-specific intrinsics. Just sign bits and
matrix multiplies. Which means SM121 support is not a research project, it is a TORCH_CUDA_ARCH_LIST="12.1" away. As usual.
Apparently patience does occasionally have a
payoff, even on Blackwell.
FP8 KV cache is the current baseline — TurboQuant slots into the same abstraction at
roughly half the bits and zero the accuracy loss. Whether that counts as progress is
left as an exercise for the reader.
ja, pytorch is … eigentlich sehr trivial. weil der algorithmus wirklich wirklich simpel ist. der rest ist dieses in vllm einbauen. flashinfer kommt damit z.b. gar nicht klar. also gleich eigenes backend, dann cuda. nunja. endless story.
SUMMARY:
TurboQuant is another example of how mathematically elegant doesn’t automatically mean fast to implement. The algorithm itself is beautifully simple — but it stores weights in unconventional bit widths, far from the 8/16/32-bit boundaries that hardware and software stacks are built around.
So the real battle isn’t with complex algorithms. It’s with interfaces and APIs that only want to hand you bits in nice, standard-sized boxes. Every additional library you pull in opens a new front — every change would need to be pushed upstream. And then you ask yourself: should you just bypass the whole stack and wire it up directly?
looks good. tokenrate is stable, the biggest issue as i mentioned before is the implementation in the vllm ecosystem. In other words: implementing TQ in vLLM and FlashInfer is a bit like being an entrepreneur in Germany. You see the problem, you solve the problem, but you can’t get it off the ground because everyone says it just can’t be done that way. FlashInfer cant imagine anything else than fp8 or fp16.
Well, and ultimately I had to — I essentially had to navigate around FlashInfer quite significantly. Yes, and that of course means even more work. Well, thanks for the highly flexible framework.
sometimes I have the feeling that both frameworks are a bit over-patched. Meaning, with every small addition — new model, new FFNs, new Algos — there’s a bit more patching going on, and in the end it’s almost non-deterministic, almost becoming a little language model itself, because the tooling capabilities aren’t universal at all. So you actually need a very specific combination of parameters to achieve anything, and when you dig into it, you find that someone just patched it that way at some point. And eventually it turns into one huge pile of patches. I wouldn’t necessarily call it a patchwork — it’s more of a patch heap. That’s a bit of a difference. So it works, but it’s not the tooling you’d imagine it to be — and this isn’t meant as a complaint, it’s just the reality when everything needs to move fast and wants to move fast. But who’s supposed to clean it up? Or better: when?
Maybe future contributions like TQ have to deliver the full stack. VLLM should decide where they want to end. Funding round or not, what it needs is architecture. And maybe some old braids cut off.
