Google Gemma 4 - It will work on DGX Spark?

[carlos.albarran.mx]

It’s Gemma going to be capable of running on DGX Spark?
Do you know about any recipe for previous versionss?

Welcome Gemma 4: Frontier multimodal intelligence on device

Google its introducing Gemma 4 — our most intelligent open models to date. Purpose-built for advanced reasoning and agentic workflows, Gemma 4 delivers an unprecedented level of intelligence-per-parameter. This breakthrough builds on incredible community momentum: since the launch of our first generation, developers have downloaded Gemma over 400 million times, building a vibrant Gemmaverse of more than 100,000 variants. We listened closely to what innovators need next to push the boundaries of AI, and Gemma 4 is our answer: breakthrough capabilities made widely accessible under an Apache 2.0 license.

Gemma 4: Byte for byte, the most capable open models

[cosinus]

Sure. Might take a few hours until support across all inference servers pop up.

Bringing AI Closer to the Edge and On-Device with Gemma 4 | NVIDIA Technical...

I already tried vLLM with latest Transformers v5.5.0 (which is required), but I failed:

Gemma 4 Models - which vLLM version? Any PRs spotted?

Maybe it’s just to early to test, but the official announcement mentions vLLM also in its list of inference servers. The blog post refers to the regular vLLM playbook which refers to a vLLM version 26.02 which won’t have Transformers v5.5.0 which is needed for Gemma4. But Transformers v5.5.0 doesn’t seem to be sufficient. I did a fresh rebuild eugr’s edition: ./build-and-copy.sh --tf5 -t eugr/vllm-node:20260402-tf5 Which pulls the v5.5.0: root@bb143232f90c:/workspace/vllm# uv pip list …

llama.cpp has added support already:

model: support gemma 4 (vision + moe, no audio) (#21309)

[carlos.albarran.mx]

Awesome!!!

Looks like gemma-4-31-b-thinking will achieve performance results that will be close to qwen3.5-397b-a17b and kimi-k2.5-thinking with minimal size.

[cosinus]

31B is a dense model. It will be not that fast on the Spark. I’m more interested in the 26B version - for the Spark at least.

[carlos.albarran.mx]

That is correct. Lets focus on 26B version. Working on replicating your results with the info you shared on the other thread.

[Digital_David]

Lets try them both :-)

[aniculescu]

We have just updated our vLLM and llama.cpp playbooks to use Gemma4 on DGX Spark. Check them out here: VLLM, Llama.cpp

[vedcsolution]

Very good, I see movement in these models, thank you.

[DannyTup]

docker run -it --gpus all -p 8000:8000 \
vllm/vllm-openai:gemma4-cu130 \
vllm serve ${HF_MODEL_HANDLE}

This command didn’t seem to work for me, I got:

danny@toad:~$ docker logs gemma4 -f                                                                                                                             
usage: vllm [-h] [-v]                                                                                                                                           
            {chat,complete,serve,launch,bench,collect-env,run-batch} ...                                                                                        
vllm: error: unrecognized arguments: serve google/gemma-4-26B-A4B-IT 

Removing vllm serve seems to have worked (at least, it’s gotten further and has stalled, which I presume is it downloading the model). I’m not sure why, since the output seems to suggest serve is a valid command 🙃

[AoE]

I hope they release bigger versions, because it looks promising for desktop cards, but underwhelming for the Spark if you’re looking for a model with intelligence scores closer to Qwen3.5-122B.

[WilliamD]

I have tool call failling hard with gemma-4-26B-A4B-it. I guess it’s chat template or tool parsing problem. I used vllm arm official image for gemma-4 by the way.

Got it right, claude set it up. Now my hermes working fine… Before no tool call was working due to the new tool call parsing :

solution

● Hermes + Gemma 4 tool calling fix

Root cause: vLLM’s --tool-call-parser pythonic extracts Gemma 4’s tool calls from the
model output but returns them as plain text in content rather than populating the
tool_calls field:

“content”: “call:web_search{query:Victor Wembanyama stats}”,
“tool_calls”:

Hermes only checks message.tool_calls — if it’s empty, it treats the response as a normal
text reply. Silent failure.

The fix — 3 files in hermes-agent:

1. environments/tool_call_parsers/pythonic_parser.py (new file)
   A parser that converts call:func_name{key:value,key2:value2} → structured
   ChatCompletionMessageToolCall objects. Registered as “pythonic” in the parser registry.

2. environments/tool_call_parsers/init.py
   One line: import and register the new parser.

3. run_agent.py (the actual CLI agent loop, ~line 7749)
   Added a fallback block just before if assistant_message.tool_calls:. When tool_calls is
   empty, it inspects content for either <tool_call> (existing Hermes/JSON format) or the
   regex call:[A-Za-z_]\w*{ (Gemma 4 pythonic format), and if matched, runs the appropriate
   parser to populate tool_calls before dispatch.

[DannyTup]

@WilliamD

I have tool call failling hard with gemma-4-26B-A4B-it

I saw [Bug]: Gemma4ToolParser.__init__() missing `tools` parameter — 400 error on tool calls · Issue #38837 · vllm-project/vllm · GitHub, it seems that the PR they landed to add support for Gemma4 was based on an older commit of vllm and missed a refactor that was made.

I don’t know how the vllm-openai:gemma4-cu130 Docker image was built or whether it’ll be updated with this.

I had a quick test in open-webui (without tools) and it didn’t seem bad from the few things I asked it. I’m a little skeptical it’ll compare with the larger models that fit on a Spark though.

[aniculescu]

Thanks for pointing that out. We will update the playbook momentarily.

[carlos.albarran.mx]

Ok please let us know.. so we can retest.

[carlos.albarran.mx]

Solved here: Gemma 4 Models - which vLLM version? Any PRs spotted? - #36 by eugr

[DannyTup]

Looks like there has also been a release (0.19.0) of vllm since (which also include sthe PR to fix tools), so the gemma-specific container is probably unnecessary. I expect vllm/vllm-openai:latest-aarch64-cu130 and vllm/vllm-openai:v0.19.0-aarch64-cu130 will also work.

Edit: No, I spoke too soon. Even though it’s in the release notes for 0.19.0, it fails for me because of the transformers version.

Edit2: I went back to vllm/vllm-openai:gemma4-arm64-cu130 (I don’t know why this one is arm and usually they use aarch in the tag.. maybe being built manually?), which seems like it was updated since the original release, so hopefully includes the tool fix. Will try that out later.

[jukrapope]

I try E4B and 31B on Ollama / VS Code and Claude Code CLI. It uses tools flawlessly.

However, 31B seems to be quite slow on Spark.

[aniculescu]

The container entrypoint is already vllm serve so it is not needed again in the docker run command. The playbook has been fixed and published

[carlos.albarran.mx]

Where I can find it?

[aniculescu]

The vLLM and llama.cpp playbooks can be found here: VLLM, Llama.cpp