Gemma4 Benchmarks on double DGX Sparks Ray Cluster and single DGX

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

Hardware

MSI DGX EdgeXpert

Spec Value
Architecture Grace Blackwell Superchip (GB10)
Unified memory 128 GB LPDDR5X
Memory bandwidth ~273 GB/s
Platform Ubuntu 24.04, aarch64
CUDA 13.0 (driver 580.142)

2 MSI DGX EdgeXpert combine a ray cluster

Docker Image

A customized image based on vllm/vllm-openai:gemma4-cu129 with the Ray package additionally installed.

Models Tested

Model Quantization Format Disk size Notes
google/gemma-4-31B-it bf16 safetensors ~62 GB Dense, reference baseline
google/gemma-4-26B-A4B-it bf16 safetensors ~49 GB MoE: 26B total / 4B active

Benchmark Results

Benchmarked with vllm benchmark.
Conditions: concurrency 1

# run benchmark from container in the dgx box
vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model google/gemma-4-26B-A4B-it \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 512 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --save-result

Peak token throughput (t/s — higher is better)

Model DGX TokenLen 128 TokenLen 512 Token Len 2048
31B bf16 RAY(2 Nodes) 7.00 6.00 6.00
31B bf16 Single Node 4.00 4.00 4.00
26B-A4B MoE RAY(2 Nodes) 32.00 31.00 31.00
26B-A4B MoE Single Node 25.00 24.00 24.00

Output token throughput (t/s — higher is better)

Model DGX Token Len 128 Token Len 512 Token Len 2048
31B bf16 RAY(2 Nodes) 5.65 5.57 5.46
31B bf16 Single Node 3.70 3.48 3.44
26B-A4B MoE RAY(2 Nodes) 26.81 27.26 26.72
26B-A4B MoE Single Node 23.42 23.24 23.00

Time To First Response (ms — lower is better)

Model DGX Token Len 128 Token Len 512 Token Len 2048
31B bf16 RAY(2 Nodes) 238.00 241.18 254.85
31B bf16 Single Node 490.57 619.68 689.76
26B-A4B MoE RAY(2 Nodes) 96.06 98.04 109.33
26B-A4B MoE Single Node 103.64 112.50 116.42

Conclusion

1.google/gemma-4-31B-it : the performance in 2 Nodes cluster is more than 50% higher than single node

2.google/gemma-4-26B-A4B-it : the performance in 2 Nodes cluster is only about 15% higher than single node

3.google/gemma-4-31B-it is too slow for DGX spark, even in 2 nodes cluster

3

Please submit your benchmarks to Spark Arena (Our community Benchmarks website): https://spark-arena.com

You can run models easily and stable with Sparkrun: GitHub - spark-arena/sparkrun: sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems · GitHub

Some gemma 4 benchmarks:

4

Thanks for sharing these results. I will move this to GB10 projects