It’s not totally new, diffusion models have been around for quite some time, they are used for image generation. As well as there were attempts at text generation with diffusion.
It’s the same MoE Gemma with different generation strategy I think. Should be faster, at the cost of quality.
Benchmarks using the unquantised model:
~/Developer$ uvx llama-benchy --base-url ``http://localhost:8000/v1`` --tg 1024 --exact-tg --latency-mode generation
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| google/diffusiongemma-26B-A4B-it | pp2048 | 666.86 ± 35.04 | 4578.19 ± 168.79 | 3080.47 ± 168.79 | 4578.19 ± 168.79 | |
| google/diffusiongemma-26B-A4B-it | tg1024 | 102.21 ± 23.44 | 336.00 ± 16.31 |
It’s fast, but for very specific use cases, definitely not for main agentic use. Looks to be working with tooling, but only for generation with no input.
Hopefully someone would figure out some sort of smart AR/DG hybrid.
| Case | prompt tok | out tok | wall (s) | tok/s | finish |
|---|---|---|---|---|---|
| tiny-in / tiny-out | 16 | 13 | 0.38 | 34.3 | stop |
| short-in / med-out | 22 | 121 | 1.39 | 87.2 | stop |
| short-in / long-out | 25 | 714 | 5.03 | 142.1 | stop |
| med-in / med-out | 782 | 72 | 0.94 | 76.6 | stop |
| long-in / short-out | 3060 | 35 | 1.18 | 29.6 | stop |
All stop, no truncation, coherent output.
| Workload | Fit | Effective tok/s |
|---|---|---|
| Long-form generation (output ≫ input) | Strong — ~6× AR | ~142 |
| Bulk drafting / summary-expansion | Strong | 87–142 |
| Short-output / long-input (e.g. recall extraction) | Poor — no AR advantage + lower quality | ~30 |
| Interactive low-latency chat | Weak — ~3s TTFR | n/a |
| Long-context (32K+) | Avoid | unstable |
Baseline: AR Gemma-4 26B-A4B ≈ 23.7 tok/s flat on the same box. DiffusionGemma wins ~6× for long outputs, converges to ~30 for short-output/long-input.
Routing implication: good for a bulk/long-form fast lane; not a fit for recall extraction (longish input, short structured output — keep on an AR model).
“Hopefully someone would figure out some sort of smart AR/DG hybrid”
Take a look at nvidia/Nemotron-Labs-Diffusion-14B · Hugging Face
Google markets “up to 4x”. Not 7x.
I’m only seeing about double, but I hope it is just a poorly optimized stack that gets better over time. I’ve seen many other models get significantly faster after release.
With the normal Gemma 4 26B QAT MTP under llama-server, I see 100 tokens per second output very consistently. If you’re only seeing a quarter of that, then something is not correct with your setup, or you’re using the 16-bit model, which I struggle to recommend when QAT is virtually indistinguishable. (If 26B A4B QAT isn’t good enough, then the 16 bit version won’t be good enough either… it’s time for a better model altogether.)
FWIW I ran some (quality) benchmarks on this, and while it was faster, some of the scores were pretty bad compared to the originals:
| name | bfcl | bigcodebench | IfEvalCode | The Agent Co |
Overall |
|---|---|---|---|---|---|
| Gemma4 31B 14h 57m, 37,212k in, 690k out |
78.0% | 72.0% | 18.0% ts: 36.0% / 38.0% |
0.0% | 46.5% |
| Gemma4 31B speculative-config=gemma-4-31B-it-assistant(4) 8h 15m, 35,847k in, 722k out |
78.0% | 70.0% | 18.0% ts: 36.0% / 40.0% |
0.0% | 46.5% |
| Gemma4 26B-A4B 9h 10m, 469,371k in, 1,009k out |
80.0% | 62.0% | 18.0% ts: 30.0% / 44.0% |
0.0% | 45.0% |
| Gemma4 31B qat-w4a16-ct speculative-config=gemma-4-31B-it-qat-q4_0-unquantized-assistant(4) 5h 19m, 113,666k in, 857k out |
78.0% | 70.0% | 16.0% ts: 32.0% / 32.0% |
0.0% | 43.8% |
| Qwen3.6 27B 39h 51m, 43,822k in, 2,398k out |
78.0% | 62.0% | 12.0% ts: 22.0% / 14.0% |
10.0% | 36.5% |
| Gemma4 Diffusion 4h 30m, 258,452k in, 1,077k out |
78.0% | 0.0% | 2.0% ts: 2.0% / 4.0% |
10.0% | 16.9% |
It’s possible it’s doing badly with tool calls, I’ll have to do some digging. But the setup was identical for all of the variations of the model.
Hi, what is Nemotron Labs Diffusion 14B?
Is it related to Gemma 4 diffusion?
No, it’s a separate effort. Combines AR and Diffusion modes and can switch on the fly.
Moved a text-only workload over and saw similar behavior as the benchmark above: good throughput for medium sized one-shot responses, meh responsiveness and hangs with tool calling.
I started digging into testing google/diffusiongemma-26B-A4B-it in vLLM today and am seeing it perform as well in tool calling scenarios as the non-diffusion Gemma4 26B-A4B. Initial BFCL (older tool calling leaderboard) eval results show it doing quite decent - better than I expected for their first diffusion model release.
Now, in vLLM this model triggers a lot of bugs in our gemma4 tool call parser when using it in streaming mode. I have an entirely new streaming parser engine PR open upstream in vLLM that first fixes issues for Qwen3. Gemma4 will be submitted as a fast followup to that once the engine + Qwen3 merges, and this should clean up the streaming tool calls for this model (and existing Gemma4 models when used with speculative decoding, like MTP, enabled). Anyone feeling adventurous can build from my branch at GitHub - bbrowning/vllm at streaming-parser-engine-2 · GitHub to get the updated qwen3 and gemma4 reasoning and tool parsers if you want to test streaming tool calls with diffusiongemma or regular Qwen3.5/3.6 or Gemma4 with MTP enabled.
Do NOT try to use this model with structured outputs in vLLM yet (including tool_choice="required"), as that frequently causes crashes when I try. This is something we’ll have to work through on the vLLM side, and not too unusual for a branch new type of model like this.
Here are the results of tool-eval-bench, skipping the categories that test structured outputs but including hardmode. 82/100 overall. These were run on an H100 because my DGX Spark was busy running other tasks today. But, just to give a baseline of model expectations.
tool-eval-bench \
--seed 42 \
--hardmode \
--parallel 2 \
--categories A B C D E F G I J K L M N P
Thank you, will pull your branch and see if it fixes the issues with streaming and tool calls
I had fixes for it in day0 community release, at least it was passing the tool-bench-eval without crashing. I believe vLLM have merged it at least partially yesterday.
Yeah, it did great in BFCL in my test (see GitHub - DanTup/spark-evals: Some benchmark results of small models and quants that fit on DGX Spark · GitHub) but really badly at the others. I don’t know if the other benchmarks use structured outputs so it might not be worth re-testing yet, but I’d definitely be interested in doing so once you think that’s fixed 🙂
This model is practically unusable. It can’t handle tool-use properly, and it’s been politically pre-trained — labeling everything as fake news over the smallest detail.
Two examples from my test:
It was supposed to translate a French X tweet. The post discussed the topic of Belfast in harsh terms. The model refused, saying it wouldn’t do it because it constitutes"hate speech." Fact is: yes, the post was written very harshly. But it’s not the model’s job to decide for me what’s good or right. It should just translate.
It was supposed to research “Fable 5.” Since it wasn’t capable enough for web search, it claimed that Fable 5 doesn’t exist. It accused me of trying to ask about fakenews. In an image analysis task (extracting text from an image), it concluded by saying that the text contained information that was “allegedly false.” That wasn’t the task.
The model is total crap!
In my case I was getting EngineDeadErrors from tool_choice=“required” tests with DiffusionGemma. Things are moving fast upstream, and I’m about a day behind at this point of my build and testing. So, it’s possible. I also see some open PRs cleaning some of this up. The general intersection of multi-token (whether via DiffusionGemma or spec decoding) and structured outputs is not in a good place generally upstream right now.
