Gemma 4 just released. Obviously Google has “collaborated” with NVIDIA on the release. Google’s announcements boasts all kinds of zero day support on multiple backends.
NVIDIA is handy with the press release RTX to Spark: Gemma 4 Accelerated for Agentic AI | NVIDIA Blog. – and if you read down the page, the best way to take advantage of this new model on your shiny spark cluster is… Ollama. Boy it’s great to be on the cutting edge again.
That’s fine, folks. You know what is working zero day? MLX.
It’s just up. :-D
Missing the announcement for vLLM, yet. But the community zero day support squad has already started its work:
Nvidia has an official NVFP4 for the 31B and suggests serving with vLLM
nvidia/Gemma-4-31B-IT-NVFP4
We have just updated our vLLM and llama.cpp playbooks to use Gemma4 on DGX Spark. Check them out here: VLLM, Llama.cpp
That’s a pleasant surprise. Thank you.
Some numbers for Ollama and Gemma4 (added Qwen3.5 to have a value to compare with):
qwen3.5:35b-a3b-q4_K_M, response token/s: 59.68
qwen3.5:122b-a10b-q4_K_M, response token/s: 24
gemma4:26b-a4b-it-q4_K_M, response token/s: 58.67
gemma4:31b-it-q4_K_M, response token/s: 10.34
Hello everyone,
I’m new to running AI models locally. I just got my hands on nvdia DGX spark. I run gemma 4 31b using Ollama. It works but is sluggish when I use it as model provider for my Hermes agent.
Are there better (more performant) ways to run Gemma 4 on DGX Spark. Or are there model variants specifically designed for GB10? E.g. larger context window, quantization techniques etc.?
Thanks for advice.
Dense models are the Spark’s kryptonite. The architecture trades off memory capacity for bandwidth. Gemma 4 26b-a4b will run much faster, as will GPT-OSS 120b, Qwen 3.5 122b, etc. Basically, MoE models are much better suited for the design of the Spark.
Been using the exact same also. It is a very goot combo. Hermes + Qwen3.5-122B Int4 on the DGX work wonders.
Thanks a lot for the tip! I’ll definitely give this a shot. What tool does it use to run the model? Is it Ollama too? From the README.md I see it uses docker. Does this imply Ollama? Does it have openai-like API built in so that I can just point Hermes to it or does it require additional setup?
It is using vLLM in a docker. The repos provide a install.sh script. It will build the docker and launch it. There is also manual steps that you can follow.