46tok/s with RedHatAI/gemma-4-26B-A4B-it-NVFP4

[say3]

| model                             |            test |             t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:----------------------------------|----------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |          pp2048 | 7296.94 ± 11.07 |              |    332.87 ± 0.43 |    280.80 ± 0.43 |    332.95 ± 0.42 |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |            tg32 |    47.49 ± 0.02 | 49.03 ± 0.02 |                  |                  |                  |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |  pp2048 @ d4096 |  6295.58 ± 7.45 |              |   1083.91 ± 5.75 |   1031.84 ± 5.75 |   1083.99 ± 5.75 |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |    tg32 @ d4096 |    46.93 ± 0.06 | 48.45 ± 0.06 |                  |                  |                  |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |  pp2048 @ d8192 |  5383.59 ± 5.05 |              |   2081.20 ± 2.45 |   2029.13 ± 2.45 |   2081.30 ± 2.44 |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |    tg32 @ d8192 |    46.17 ± 0.07 | 47.66 ± 0.07 |                  |                  |                  |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 | pp2048 @ d16384 |  4081.53 ± 9.25 |              |  4935.81 ± 19.58 |  4883.74 ± 19.58 |  4935.89 ± 19.58 |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |   tg32 @ d16384 |    45.27 ± 0.15 | 46.74 ± 0.16 |                  |                  |                  |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 | pp2048 @ d32768 |  2627.16 ± 5.73 |              | 14377.94 ± 67.04 | 14325.87 ± 67.04 | 14378.03 ± 67.04 |
| RedHatAI/gemma-4-26B-A4B-it-NVFP4 |   tg32 @ d32768 |    43.60 ± 0.33 | 45.02 ± 0.34 |                  |                  |                  |

llama-benchy (0.3.5)
date: 2026-04-08 22:05:18 | latency mode: generation

Steps:

1.

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./build-and-copy.sh --tf5 --apply-vllm-pr 39045

2. Modify ./spark-vllm-docker/recipes/gemma4-26b-a4b.yaml

3. # Recipe: Gemma4-26B-A4B
   # Gemma4-26B-A4B model in online FP8 quantization
   
   recipe_version: "1"
   name: Gemma4-26B-A4B
   description: vLLM serving Gemma4-26B-A4B
   
   # HuggingFace model to download (optional, for --download-model)
   model: RedHatAI/gemma-4-26B-A4B-it-NVFP4
   
   # Only cluster is supported
   cluster_only: false
   solo_only: false
   
   # Container image to use
   container: vllm-node-tf5
   
   build_args:
     - --tf5
   
   # Mods
   mods:
     - mods/fix-gemma4-tool-parser
   
   # Default settings (can be overridden via CLI)
   defaults:
     port: 8000
     host: 0.0.0.0
     tensor_parallel: 2
     gpu_memory_utilization: 0.7
     max_model_len: 262144
     max_num_batched_tokens: 8192
   
   # Environment variables
   env: {}
   
   # The vLLM serve command template
   command: |
     vllm serve RedHatAI/gemma-4-26B-A4B-it-NVFP4  \
       --max-model-len {max_model_len} \
       --gpu-memory-utilization {gpu_memory_utilization} \
       --port {port} \
       --host {host} \
       --load-format fastsafetensors \
       --enable-prefix-caching \
       --enable-auto-tool-choice \
       --tool-call-parser gemma4 \
       --reasoning-parser gemma4 \
       --kv-cache-dtype fp8 \
       --max-num-batched-tokens {max_num_batched_tokens} \
       -tp {tensor_parallel} --distributed-executor-backend ray
   

4. ./spark-vllm-docker/run-recipe.sh gemma4nvfp4 –solo -d

[raphael.amorim]

Hey, feel free to submit your benchmark to https://spark-arena.com

we don’t have any gemma-4-26B-A4B-it-NVFP4 contributions yet!

[vgoklani]

Hi @raphael.amorim The leaderboard should have evals too. For example, some of the quantizations i’ve seen for this particular model have very noisy reasoning traces - that implies poor quantizations. There is no point in being fast if the output is unsatisfactory.

[Digital_David]

Is this one of them>?

[say3]

I don’t have linkedin account sorry

[raphael.amorim]

We’re still restricting anonymous contributions, but we will get there. We’re putting the tools in place on Sparkarena and Sparkrun for that

[djordjestojanovic1992]

That would be huge, only then could we see how true some of the claimed improvements here on the forum actually are, please implement this. @raphael.amorim - I don’t even have a Spark, getting mine in 2 days and can’t wait to try, would be very willing to help if I can and if I may with trying out different versions of models, if Eval gets implemented. :) Thanks for the great work.

[raphael.amorim]

That’s in the works @vgoklani, but those take a long time to run and we only have a few Sparks. After we standardize the execution models. We’re going to add the functionality for users to run locally and distribute as well through Sparkrun, so users an update their existing and new recipes with intelligence benchmarks as well.

[Digital_David]

@say3

./build-and-copy.sh --tf5 --apply-vllm-pr 39045

Looks like vllm-pr 39045 was merged in yesterday, so --apply-vllm-pr 39045 is no longer needed if you update vllm

Applying --apply-vllm-pr 39045 causes the build to fail:

0.713 error: patch failed: vllm/model_executor/models/gemma4.py:1248
0.713 error: vllm/model_executor/models/gemma4.py: patch does not apply

ERROR: failed to build: failed to solve: process “/bin/sh -c if [ -n "$VLLM_PRS" ]; then echo "Applying PRs: $VLLM_PRS"; for pr in $VLLM_PRS; do echo "Fetching and applying PR #$pr…"; curl -fL "Sign in to GitHub · GitHub” | git apply -v; done; fi" did not complete successfully: exit code: 1

But some other error is causing it to not load :-(

[vgoklani]

Thanks @raphael.amorim looking forward to it.

[say3]

I’m using spark-vllm-docker last build.

[Digital_David]

I am as well I did this 1st:

cd /spark-vllm-docker
git pull
rm wheels/.whl wheels/.-commit
./build-and-copy.sh -t vllm-node-tf5 --rebuild-vllm

1st run ./run-recipe.sh gemma4-26b-a4b-nvfp4 --solo --setup
2nd run ./run-recipe.sh gemma4-26b-a4b-nvfp4 --solo

Then crash.

@eugr can you give this one a try, or did I build it incorrectly?

[say3]

./build-and-copy.sh --tf5 --apply-vllm-pr 39045

spark-vllm-docker not updated new vllm.

[Digital_David]

See previous message,

@say3

./build-and-copy.sh --tf5 --apply-vllm-pr 39045

See previous message about vllm-pr 39045 now merged and crashes vllm-builder

=> ERROR [vllm-builder 4/8] RUN if [ -n “39045” ]; then echo "Applying PRs: 39 0.8s

[vllm-builder 4/8] RUN if [ -n “39045” ]; then echo “Applying PRs: 39045”; for pr in 39045; do echo “Fetching and applying PR #$pr…”; curl -fL “https://github.com/vllm-project/vllm/pull/${pr}.diff” | git apply -v; done; fi:

[say3]

Maybe re-clone the repo

[say3]

–tf5 not equal -t vllm-node-tf5

[eugr]

There were some issues with the build pipeline when I was traveling, it’s been fixed now.
BTW, --tf5 without specifying the image name will now produce vllm-node-tf5 by default ;)

[giraudremi92]

COOL ! @eugr are you working on the gemma4 mtp addon?

[eugr]

Yes, I plan to test that one.