Has anyone actually succeeded in deploying gemma4 dense using both the new MTP AND turboquant?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I spent the whole day Today with gemini pro back-forth to try and implement the new MTP and turboquant for gemma-4 to no avail.. Gemini threw me the rabbit hole of trying to compile mitkox approach from scratch and every dockerfile compilation would take anywhere from 30-120 minutes depending on the iteration. I lost hope…

Has anyone actually successfully deployed Gemma4-31B FP8 using both google/gemma-4-31B-it-assistant AND turboquant? I find it abysmal that when I load bare bones Gemma4-31B FP8 on my Spark it gets 4-6 t/s decode and only 150k tokens context on 0.91 utilization whilst my Qwen 3.6 MOE gives me 30-40 t/s decode AND 1.5M context. It just sounds like a holy grail to achieve both 2-3X improvement in t/s AND 3-6X increase in token context using both.

2

With llama.cpp, I can fit the 8-bit Gemma 4 31B model with its full 262000 token context window at full fp16 KV cache in about 71GB of memory.

You should not need turboquant or any other form of KV cache quant to fit that model with 128GB of memory, even with MTP, since Gemma 4’s MTP is uniquely designed to reuse the target model’s KV cache.

I have not had the motivation to bother with running Gemma 4 under vLLM since vLLM is always such a pain.

3

why do you run the model in 8bit, but the kv-cache in FP16?

4

KV cache accuracy seems more important than the accuracy of the model weights, and high accuracy KV cache doesn’t affect performance like higher precision weights (which can be huge). I could fit the model weights into memory as fp16, but then the model would just be extremely slow.

Twice the size of the model weights = half the performance, basically. The same logic doesn’t apply to the KV cache.

5

You do not need TurboQuant to have plenty of performance and cache with Gemma4-31b-it on this hardware.

I’ll cross-post this for you - strongly encourage you to try the recipe with this particular NVFP4A16 quant. The MTP is a big deal.

30-40 tok/s for most workloads. Floor is about 19, still quite usable, even on the hardest document analysis. 1.2 million tokens in KV cache.

Rebuild spark-vllm-docker with --tf5 --rebuild-vllm and use this startup script:

#!/bin/bash
~/containers/spark-vllm-docker/launch-cluster.sh -t vllm-node-tf5 --solo -d \
  exec vllm serve ebircak/gemma-4-31B-it-4bit-NVFP4A16-GPTQ \
  --served-model-name Gemma4-31b-it \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --port 8000 \
  --host 0.0.0.0 \
  --max-num-seqs 4 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --load-format instanttensor \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --speculative-config '{"method":"mtp","model":"google/gemma4-31b-it-assistant","num_speculative_tokens":7}'

Benchmarks and other thoughts in the other thread here Gemma4 draft models are now available - #8 by joshua.dale.warner

6

I have made some progress! I’m now testing a full fp16 model and MTP. For some reason (probably good reason, that I haven’t had time to fully understand yet) the MTP also seems to be doing wonders for kvcache.

Using nightly and that’s it.

docker run -itd \
  --name gemma4-server \
  --ipc=host \
  --network host \
  --shm-size 16G \
  --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /dev/shm:/dev/shm \
  -e HF_TOKEN="********" \
  -e HF_HUB_ENABLE_HF_TRANSFER=1 \
  vllm/vllm-openai:nightly \
  --model google/gemma-4-31B-it \
  --served-model-name gemma4-dense-MTP \
  --tensor-parallel-size 1 \
  --dtype auto \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 6 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --max-num-batched-tokens 32768 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}' \
  --port 8000

I’m getting: GPU KV cache size: 517,090 tokens

Maximum concurrency for 262,144 tokens per request: 1.97x

Getting 8-12 t/s which is still better than vanilla, but nowhere near 30/40 from the other post (I also saw it). I’m just a little weary of using NVFP4 quants for the accuracy loss. Although it’s probably worth giving it a shot.