Has anyone actually succeeded in deploying gemma4 dense using both the new MTP AND turboquant?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
5

You do not need TurboQuant to have plenty of performance and cache with Gemma4-31b-it on this hardware.

I’ll cross-post this for you - strongly encourage you to try the recipe with this particular NVFP4A16 quant. The MTP is a big deal.

30-40 tok/s for most workloads. Floor is about 19, still quite usable, even on the hardest document analysis. 1.2 million tokens in KV cache.

Rebuild spark-vllm-docker with --tf5 --rebuild-vllm and use this startup script:

#!/bin/bash
~/containers/spark-vllm-docker/launch-cluster.sh -t vllm-node-tf5 --solo -d \
  exec vllm serve ebircak/gemma-4-31B-it-4bit-NVFP4A16-GPTQ \
  --served-model-name Gemma4-31b-it \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --port 8000 \
  --host 0.0.0.0 \
  --max-num-seqs 4 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --load-format instanttensor \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --speculative-config '{"method":"mtp","model":"google/gemma4-31b-it-assistant","num_speculative_tokens":7}'

Benchmarks and other thoughts in the other thread here Gemma4 draft models are now available - #8 by joshua.dale.warner