nvidia/Gemma-4-26B-A4B-NVFP4

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

What’s the best way to run this model across two sparks:

we would like to use expert-parallelisim

Is there an optimized flash-infer path for vLLM? How do we incorporate the latest MoE fixes for nvFP4 from nvidia? Thanks!

2

Such a small model is not going to make sense to split over two Sparks. The overhead of communication will be a lot.

3

it should be fine with expert parallelism. gpt-oss-120b does really well with 2 sparks, it’s at the top of the leaderboard.

4

Tensor-parallel will give better performance gains than expert parallel in a Spark cluster…

5

taking something that runs perfectly fine on a single node does not always give you the return on splitting it across 2 nodes, especially when that single node already excels at running in parallel.

the only way I could see this maybe working better is if you shard the experts across the nodes? But even then the work to do it vs the returns, questionable IMO

6

We’d prefer to test. As mentioned above, gpt-oss-120b works well on 2 sparks…

Is there an optimized docker image that we can just pull with the latest fixes for flashinfer?

We’d like to test this: https://github.com/vllm-project/vllm/pull/40082#issuecomment-4349406309

cc: @eugr

7

b12x backend doesn’t support all models yet, definitely not Gemma (it’s not even supported by standard Flashinfer backend, but works with vLLM CUTLASS):

./launch-cluster.sh -t vllm-node-tf5 --solo \
  exec vllm serve nvidia/Gemma-4-26B-A4B-NVFP4 \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --enable-auto-tool-choice \
  --trust-remote-code \
  --port 8888 \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.85 \
  --max-num-batched-tokens 32768 \
  --enable-prefix-caching
model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
nvidia/Gemma-4-26B-A4B-NVFP4 pp2048 2723.35 ± 649.96 814.11 ± 221.92 805.02 ± 221.92 814.11 ± 221.92
nvidia/Gemma-4-26B-A4B-NVFP4 tg32 29.40 ± 0.47 29.67 ± 0.47
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_pp @ d4096 4489.98 ± 513.16 1023.63 ± 104.63 1014.55 ± 104.63 1023.63 ± 104.63
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_tg @ d4096 29.68 ± 1.40 30.23 ± 1.73
nvidia/Gemma-4-26B-A4B-NVFP4 pp2048 @ d4096 3594.97 ± 64.87 578.95 ± 10.30 569.87 ± 10.30 578.95 ± 10.30
nvidia/Gemma-4-26B-A4B-NVFP4 tg32 @ d4096 28.65 ± 0.04 29.00 ± 0.00
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_pp @ d16384 3819.19 ± 80.30 4679.28 ± 88.24 4670.20 ± 88.24 4679.28 ± 88.24
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_tg @ d16384 28.30 ± 0.67 28.67 ± 0.94
nvidia/Gemma-4-26B-A4B-NVFP4 pp2048 @ d16384 1994.97 ± 13.63 1035.71 ± 6.99 1026.63 ± 6.99 1035.71 ± 6.99
nvidia/Gemma-4-26B-A4B-NVFP4 tg32 @ d16384 27.94 ± 0.16 28.67 ± 0.47
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_pp @ d32078 2539.93 ± 21.07 13726.55 ± 116.16 13717.47 ± 116.16 13726.55 ± 116.16
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_tg @ d32078 28.16 ± 0.97 28.67 ± 0.94
nvidia/Gemma-4-26B-A4B-NVFP4 pp2048 @ d32078 1276.25 ± 13.26 1613.95 ± 16.68 1604.87 ± 16.68 1613.95 ± 16.68
nvidia/Gemma-4-26B-A4B-NVFP4 tg32 @ d32078 27.48 ± 0.10 28.00 ± 0.00
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_pp @ d65535 1484.13 ± 1.42 48078.91 ± 49.94 48069.83 ± 49.94 48078.91 ± 49.94
nvidia/Gemma-4-26B-A4B-NVFP4 ctx_tg @ d65535 27.39 ± 0.73 28.33 ± 0.47
nvidia/Gemma-4-26B-A4B-NVFP4 pp2048 @ d65535 727.30 ± 7.54 2825.28 ± 29.29 2816.20 ± 29.29 2825.28 ± 29.29
nvidia/Gemma-4-26B-A4B-NVFP4 tg32 @ d65535 26.97 ± 0.22 28.00 ± 0.00

llama-benchy (0.3.7)
date: 2026-05-01 13:31:39 | latency mode: api

Wasn’t able to launch this particular model in the cluster though.

8

thanks @eugr !