Community Docker adds support for DiffusionGemma

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Another day0 release :)

This time it’s Diffusion Gemma from Google AI. Read NVIDIA Tech Blog for more details.

Model card: google/diffusiongemma-26B-A4B-it · Hugging Face

Community docker: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub

Added four solo-only DiffusionGemma recipes:

  • diffusion-gemma-bf16-thinking for google/diffusiongemma-26B-A4B-it with thinking enabled.
  • diffusion-gemma-bf16 for google/diffusiongemma-26B-A4B-it with thinking disabled.
  • diffusion-gemma-nvfp4-thinking for nvidia/diffusiongemma-26B-A4B-it-NVFP4 with thinking enabled.
  • diffusion-gemma-nvfp4 for nvidia/diffusiongemma-26B-A4B-it-NVFP4 with thinking disabled.

The non-thinking variants still keep --reasoning-parser gemma4, since these models can emit Gemma4 channel markers even when thinking is disabled.

Example:

./hf-download.sh google/diffusiongemma-26B-A4B-it
./run-recipe.sh diffusion-gemma-bf16-thinking --solo

UPDATE: NVFP4 checkpoint is live now too, run with:

./hf-download.sh nvidia/diffusiongemma-26B-A4B-it-NVFP4
./run-recipe.sh diffusion-gemma-nvfp4-thinking --solo
2

Please don’t run llama-benchy to test performance yet - I’ll be pushing a new version soon that correctly supports diffusion model generation quirks.

3

Thank you so much, as always!
Thanks to your updates, I’m really able to make great use of my three DGX Spark systems.
I truly appreciate everything you do.
I’m always rooting for your continued work!😊

4

New version of llama-benchy is out, but a few important things to know:

  • This model emits results in 256 token blocks by default, even if response is shorter, so you need to generate > 256 tokens to get a better sense of speed.
  • I suggest using --exact-tg parameter to clamp the output to desired value.
  • This model has slower prefill speeds, but much-much faster token generation, so more tokens you generate, larger the performance boost compared to autoregressive version.
  • Since it generates many tokens at once, the performance is very sensitive to the input prompt, just like MTP, so benchmarks may not be representative for your specific workflow.

Example:

uvx llama-benchy --base-url http://localhost:8000/v1 --tg 1024 --exact-tg --latency-mode generation
5

NVFP4 checkpoint is live now too, run with:

./hf-download.sh nvidia/diffusiongemma-26B-A4B-it-NVFP4.
./run-recipe.sh diffusion-gemma-nvfp4-thinking --solo
6

Amazing! I think we are all very lucky to have you 😀

Probably a silly question, but do your new recipe updates make their way straight into sparkrun? That tends to be my typical model serving route now.

7

Yes. @eugr’s repo is one of the officially registered sparkrun registries, so that means that you can run things from his repo via sparkrun!

If you reference recipes via sparkrun run @eugr/<recipe-name>, it should automatically download it if missing from your local offline recipe cache.

You can also type sparkrun update which will ensure that you have the latest sparkrun AND ensure that your local recipes are up-to-date (meaning it’ll pull new recipes from @eugr’s repo)

8

Thanks so much for the speedy response. You guys rock!

9

Awesome i ll test it !!! thanks @eugr_nv

10

is this one is ok for the recipes ?

11

Haven’t tested anything other than the two linked in the original post.

12
model test t/s (total) t/s (req) peak t/s peak t/s (req) ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp5000 (c1) 1410.49 ± 114.51 1410.49 ± 114.51 3573.94 ± 305.68 3569.91 ± 305.68 3573.94 ± 305.68
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg512 (c1) 103.36 ± 20.55 103.36 ± 20.55 359.00 ± 20.61 359.00 ± 20.61
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp5000 (c5) 1587.93 ± 142.26 385.38 ± 72.02 13391.86 ± 2248.18 13387.82 ± 2248.18 13391.86 ± 2248.18
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg512 (c5) 106.15 ± 7.75 45.99 ± 22.19 814.33 ± 122.51 301.47 ± 22.13

First llama benchy on the spark , nvfp4 model (thinking recipe)
Prefill is linear , i will try to change that to upgrade ttft

13

Make sure you are using the latest llama-benchy and better clamp with --exact-tg as the diffusion model always generates full canvas even if it’s half-empty.

14

Yes I did it with your command and new install

But prefill seems slow on the benchmark

15

Yes, I mentioned it in the post.

Not sure how accurately I can measure it with llama-benchy though, because this model works a bit differently. vLLM logs completely different prefill numbers in the logs, but they are aggregate in the batch, so not representative either.

16

Tested nvidia/diffusiongemma-26B-A4B-it-NVFP4 and tg1024 for now, results looks good, although there is some fluctuation between runs:

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp2048 1206.87 ± 229.94 2626.66 ± 344.47 1762.28 ± 344.47 2626.66 ± 344.47
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg1024 133.13 ± 20.75 374.33 ± 7.04
model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp2048 1302.19 ± 488.64 2667.53 ± 526.34 1771.18 ± 526.34 2667.53 ± 526.34
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg1024 162.94 ± 43.94 443.33 ± 73.74
model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp2048 1022.30 ± 105.80 3044.38 ± 195.99 2024.92 ± 195.99 3044.38 ± 195.99
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg1024 171.91 ± 62.98 438.00 ± 96.29
model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp2048 1203.53 ± 82.93 2701.20 ± 120.49 1710.79 ± 120.49 2701.20 ± 120.49
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg1024 117.21 ± 15.31 367.33 ± 6.94
model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/diffusiongemma-26B-A4B-it-NVFP4 pp2048 1177.93 ± 161.17 2635.65 ± 221.11 1769.46 ± 221.11 2635.65 ± 221.11
nvidia/diffusiongemma-26B-A4B-it-NVFP4 tg1024 251.56 ± 111.92 505.33 ± 121.08

In the vLLM log I see token generation throughput varying from 102 t/s up to 179.2. Looks very impressive, I’ll test with real (non-aligned) payloads.

17

I will test it in real word conditions with some rag and see how it performs :)

18

That’s the way! Please post your impressions!

19

Is that a totally new architecture again by Google? Mega mtp embedded into the model arch. If it takes off it will be insane. QAT quant by Google are already showing better results at q4 than a normal q6 quant.

20

TBH numbers here are sorta underwhelming (or maybe i’m reading them wrong). Spark Arena’s “normal” Gemma 26 clocks at 100 tok/s, it’s not 7x jump for diffusion (especially considering quality is worse), as marketed by Google.

Still neat.