Has anyone tried out the new google/gemma-4-12B-it · Hugging Face model? I was able to easily load on a single Spark and very early use has been very positive, getting around 30 TPS generation.
MODEL DOWNLOADS (~24GB total):
Main model (BF16, \~23GB)
huggingface-cli download google/gemma-4-12b-it \\
--local-dir /models/gemma-4-12B-it
Draft model for speculative decoding (\~838MB)
huggingface-cli download google/gemma-4-12b-it-assistant \\
--local-dir /models/gemma-4-12B-it-assistant
DOCKER IMAGE:
docker pull vllm/vllm-openai:gemma4-unified
(\~29.8GB --- includes CUDA 13.0, vLLM with Gemma 4 support)
RUN COMMAND:
docker run -d \\
--name gemma4-12b \\
--gpus all \\
--restart unless-stopped \\
-v /models/gemma-4-12B-it:/model \\
-v /models/gemma-4-12B-it-assistant:/draft-model \\
vllm/vllm-openai:gemma4-unified \\
/model \\
--served-model-name gemma-4-12b \\
--max-model-len 262144 \\
--max-num-seqs 10 \\
--gpu-memory-utilization 0.85 \\
--kv-cache-dtype fp8 \\
--enable-auto-tool-choice \\
--reasoning-parser gemma4 \\
--tool-call-parser gemma4 \\
--speculative-config '{"model": "/draft-model", "num_speculative_tokens": 5}' \\
--host 0.0.0.0 \\
--port 8000
Yes, in Hermes with Mimo 2.5 pro as the orchestrator and qwen3.6 35b and minimax 2.7 running on my other 3 GB10s… the 4 models together work beautifully and cost is super minimal with mimo (which btw is pretty magical especially for the price).
Nice, thanks for sharing… wish I’d found this earlier while testing Gemma 4 12B on Spark. Same experience here: MTP is what made it feel usable.
num_speculative_tokens=5 worked well for me too. I ended up settling on gpu-memory-utilization=0.75, which still gets me ~29 tok/s while leaving room for NemoHermes and Open WebUI alongside it.
Hermes user here as well. I have a more general question. How often is Hermes calling its sub agent (or 2nd LLM) for you guys and what are you doing that triggers it? I’ve been running Hermes for a long time now and only once has it ever tried to call a secondary model. I use it to do security and patching of my computers in my home, it helped me design and build my home lab, it manages the home lab in terms of care and feeding of all vm’s / hosts, as well as pentesting. I use it for email and to-do’s. Never seems to want or “need” to call another model. It also built my entire github-like repo (Forejo) and uses it to store files such as the terraform and ansible files it uses for automated deployment of VMs ect..
Full disclosure I read the Hermes documentation, at the time it was quite limited. The “video” was nothing useful. So I dont know if I’m just using it wrong or what. It had deepseek 6B as its backup deep thinker model. But I got rid of it because it was never being used. I just have one model a highly tuned qwen 3.6 26B FP8 that runs amazingly well, a tad slow but not horrendous.
Just stuff Soul MD with instruction to delegate if it has any pieces of work that can be done in parlel. It works. But if you give it piecemeal tasks it has no reason to use subagents.
I wanted to try the unquantized version first before I moved down the line - also, I tried this yesterday before there were any more efficient quants/recipes out for vLLM which I’m sure by today there are (haven’t checked yet)
Can you share your recipe? Have you notice loops or tools call problem on large context with your recipe maybe? For my all tested qwen3.6 26B are amazing… till large context kick in, then some strange things start to appear here and there… I tried a lot of them from good souls from this community.
Yeah, it’s actually pretty good model, fared slightly above my trusted qwen 3.5 9b in quantitative trading and cta series 3 exams tests :) I am selecting a model for sft and Lora. Qwen is a speed monster with thinking off, 90 t/s on 5070ti, 60-65 t/s on spark. This one was 60, prob 45-50 on spark without mtp.
Ps it’s a small model, if you don’t run high concurrency you can definitely use it on llamacpp in q6 faster than on vllm, trusty bartowski high quality quants