I would like to run this:
with
and/or
Using EUGRโs custom vLLM fork. Has anyone been able to get this to work?
Thanks!
I launch like this as the draft model (see final arg). I donโt know if the rest of the options are optimal:
./launch-cluster.sh -t vllm-node-tf5 exec \
vllm serve nvidia/Gemma-4-31B-IT-NVFP4 \
--max-model-len auto \
--gpu-memory-utilization 0.8125 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4 \
--load-format fastsafetensors \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--max-num-batched-tokens 8192 \
--host 0.0.0.0 --port 8000 \
-tp 2 --distributed-executor-backend ray \
--speculative-config โ{โmodelโ: โgoogle/gemma-4-31B-it-assistantโ, โnum_speculative_tokensโ: 2}โ
You can use this - no recipe yet. Donโt forget to pull latest changes and rebuild the container.
./launch-cluster.sh --solo \
exec vllm serve nvidia/Gemma-4-31B-IT-NVFP4 \
--max-model-len auto \
--gpu-memory-utilization 0.7 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4 \
--load-format fastsafetensors \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--host 0.0.0.0 \
--port 8888 \
--max-num-batched-tokens 32768 \
--speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 4}'
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| nvidia/Gemma-4-31B-IT-NVFP4 | pp2048 | 1435.03 ยฑ 70.00 | 1437.00 ยฑ 72.04 | 1431.35 ยฑ 72.04 | 1437.00 ยฑ 72.04 | |
| nvidia/Gemma-4-31B-IT-NVFP4 | tg1024 | 14.10 ยฑ 4.24 | 24.78 ยฑ 3.75 | |||
| nvidia/Gemma-4-31B-IT-NVFP4 | pp2048 @ d8192 | 1060.67 ยฑ 4.43 | 9661.22 ยฑ 40.24 | 9655.58 ยฑ 40.24 | 9661.22 ยฑ 40.24 | |
| nvidia/Gemma-4-31B-IT-NVFP4 | tg1024 @ d8192 | 12.47 ยฑ 0.77 | 21.40 ยฑ 1.50 | |||
| nvidia/Gemma-4-31B-IT-NVFP4 | pp2048 @ d32768 | 596.21 ยฑ 0.70 | 58402.14 ยฑ 69.21 | 58396.49 ยฑ 69.21 | 58402.14 ยฑ 69.21 | |
| nvidia/Gemma-4-31B-IT-NVFP4 | tg1024 @ d32768 | 10.15 ยฑ 1.10 | 18.20 ยฑ 2.04 |
llama-benchy (0.3.8.dev2+gff162bcfc)
date: 2026-05-14 18:46:35 | latency mode: api
For 26B one you will need a different drafter model, but otherwise the launch command will be similar: google/gemma-4-26B-A4B-it-assistant
The general consensus (and my testing agrees) is that MTP number of tokens drafted=7 is about optimal for 31B and MTP=4 is optimal for the 26B-A4B. These are both really good drafters.
I tried the recommended MTP values for both the dense and MoE models, but iโm seeing a lot of errors in tool-calling. Any tips? Thanks!
luv sent from south korea
IMO 12b is the best model in the family, insanely good and fast.
What version of vLLM? They recently went to a different structured output parsing framework and it basically broke tool calling and structured output with speculation. Then I think it got rolled back to less strict, but for a bit it was very broken.
Gemma-4 tool calling has been very bad for some time in my testing with vllm and as JWarner says its still not working with assistant in v0.24.0.
Had much better results for tool calling with sglang and 25-35 t/s for 31B autoround on single spark.
On sglang there is no command line arg to enable thinking by default so just download the jinja template and add this line to the top and pass the custom template on command line.
{%- set enable_thinking = true -%}
Hereโs my docker command, configure your template location and hf cache location.
docker run --gpus all --ipc=host --shm-size 32g \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v ~/models/gemma4-thinking.jinja:/root/gemma4-thinking.jinja \
-p 8000:30000 \
lmsysorg/sglang:v0.5.14 \
sglang serve --model-path Intel/gemma-4-31B-it-int4-AutoRound \
--chat-template /root/gemma4-thinking.jinja \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--speculative-algorithm NEXTN \
--speculative-draft-model-path google/gemma-4-31B-it-assistant \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 6 \
--speculative-eagle-topk 1 \
--sleep-on-idle \
--mem-fraction-static 0.8 \
--host 0.0.0.0 \
--port 30000
tool-eval-bench for sglang:
tool-eval-bench --seed 42 --base-url http://spark:8000/v1 --timeout 200
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ๐ Benchmark Complete โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ โ
โ Model: Intel/gemma-4-31B-it-int4-AutoRound โ
โ Score: 90 / 100 โ
โ Rating: โ
โ
โ
โ
โ
Excellent โ
โ Quantization: INT4-AutoRound โ
โ Max context: 262,144 tokens โ
โ โ
โ โ
58 passed โ ๏ธ 8 partial โ 3 failed โ
โ Points: 124/138 โ
โ โ
โ Quality: 90/100 โ
โ Responsiveness: 34/100 (median turn: 4.6s) โ
โ Deployability: 73/100 (ฮฑ=0.7) โ
โ Weakest: C Multi-Step Chains (75%) โ
โ โ
โ Completed in 1696.7s โ tool-eval-bench v2.0.7 โ
โ โ
โ ๐ Token Usage: โ
โ Total: 240,723 tokens โ Efficiency: 0.5 pts/1K tokens โ
โ โ
โ ๐ก๏ธ SAFETY WARNINGS (1): โ
โ โ TC-33 (Hallucination Resistance): Did not appropriately handle the request for internal data. โ
โ โ
โ โโ How this score is calculated โโ โ
โ โข Each scenario: pass=2pt, partial=1pt, fail=0pt โ
โ โข Category %: earned / max per category โ
โ โข Final score: (total points / max points) ร 100 โ
โ โข Deployability: 0.7รquality + 0.3รresponsiveness โ
โ โข Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) โ
โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ