How to run Gemma-4-NVFP4 in vLLM Docker?

[tho.nguyenvan]

This is my first time using Ascent GX10.

On Hugging Face, there is already an NVFP4 model for Gemma-4 (nvidia/Gemma-4-31B-IT-NVFP4).

How can I deploy it on Docker using vLLM?

Thank you everyone for listening.

[grindstone]

i am playing with latest https://github.com/johnnynunez/vllm/commits/main/

eugr repo seems to have something something for nemotron nvfp4 that might be useful to play with else wait for offical support there.

Nothing wrong going down the rabbit hole with claude and that link johnny repo either for the “experience” :)

[blainesworld]

If you haven’t already, I would take a look at the messages in this topic

Gemma 4 Models - which vLLM version? Any PRs spotted?

Maybe it’s just to early to test, but the official announcement mentions vLLM also in its list of inference servers. The blog post refers to the regular vLLM playbook which refers to a vLLM version 26.02 which won’t have Transformers v5.5.0 which is needed for Gemma4. But Transformers v5.5.0 doesn’t seem to be sufficient. I did a fresh rebuild eugr’s edition: ./build-and-copy.sh --tf5 -t eugr/vllm-node:20260402-tf5 Which pulls the v5.5.0: root@bb143232f90c:/workspace/vllm# uv pip list …

Regarding NVFP4, while it’s supported, we still don’t have great performance with it. Lots of work to do. FP8 or INT4 will likely perform better until the issues are sorted

[eugr]

Our community Docker should be able to run that NVFP4 model just fine, however since it’s a dense model, I’d recommend to stick to 26B one instead, and we have a recipe for it: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub

Having said that, I’m downloading that NVFP4 model now too.

[eugr]

So, you can run with the most recent build from the community Docker. Don’t forget to pull and rebuild the TF5 image (git pull && ./build-and-copy.sh -t vllm-node-tf5 --tf5).

./launch-cluster.sh --solo -t vllm-node-tf5 exec vllm serve nvidia/Gemma-4-31B-IT-NVFP4  \
 --max-model-len auto  \
 --gpu-memory-utilization 0.7 \
 --enable-auto-tool-choice \
 --tool-call-parser gemma4 \
 --reasoning-parser gemma4 \
 --load-format fastsafetensors \
 --kv-cache-dtype fp8 \
 --enable-prefix-caching \
 --host 0.0.0.0 \
 --port 8888

But the performance is meh, as based on the size, most of the weights weren’t really quantized to 4 bits.

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
nvidia/Gemma-4-31B-IT-NVFP4	pp2048	1550.72 ± 69.79		1332.89 ± 61.51	1324.09 ± 61.51	1333.08 ± 61.55
nvidia/Gemma-4-31B-IT-NVFP4	tg32	6.78 ± 0.00	7.00 ± 0.00
nvidia/Gemma-4-31B-IT-NVFP4	ctx_pp @ d8192	1399.04 ± 8.88		6358.06 ± 59.37	6349.26 ± 59.37	6358.19 ± 59.39
nvidia/Gemma-4-31B-IT-NVFP4	ctx_tg @ d8192	6.70 ± 0.07	7.33 ± 0.47
nvidia/Gemma-4-31B-IT-NVFP4	pp2048 @ d8192	942.13 ± 5.25		2182.66 ± 12.16	2173.87 ± 12.16	2182.83 ± 12.11
nvidia/Gemma-4-31B-IT-NVFP4	tg32 @ d8192	6.66 ± 0.02	7.00 ± 0.00
nvidia/Gemma-4-31B-IT-NVFP4	ctx_pp @ d32768	655.65 ± 3.69		54403.16 ± 306.69	54394.37 ± 306.69	54403.70 ± 306.74
nvidia/Gemma-4-31B-IT-NVFP4	ctx_tg @ d32768	6.40 ± 0.01	7.00 ± 0.00
nvidia/Gemma-4-31B-IT-NVFP4	pp2048 @ d32768	347.67 ± 0.83		5899.40 ± 14.11	5890.60 ± 14.11	5899.58 ± 14.14
nvidia/Gemma-4-31B-IT-NVFP4	tg32 @ d32768	6.40 ± 0.02	7.00 ± 0.00

llama-benchy (0.3.6.dev12+g5e7b509cb)
date: 2026-04-03 16:33:03 | latency mode: api | pp basis: ttfr

[eugr]

Intel Autoround quant is only marginally better on a single Spark (and worse pp):

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
Intel/gemma-4-31B-it-int4-AutoRound	pp2048	822.54 ± 16.25		2497.11 ± 49.35	2491.64 ± 49.35	2497.32 ± 49.37
Intel/gemma-4-31B-it-int4-AutoRound	tg32	11.02 ± 0.04	11.67 ± 0.47
Intel/gemma-4-31B-it-int4-AutoRound	ctx_pp @ d8192	744.02 ± 4.33		12000.19 ± 77.94	11994.72 ± 77.94	12000.46 ± 77.87
Intel/gemma-4-31B-it-int4-AutoRound	ctx_tg @ d8192	10.64 ± 0.01	11.00 ± 0.00
Intel/gemma-4-31B-it-int4-AutoRound	pp2048 @ d8192	617.18 ± 9.65		3324.61 ± 52.49	3319.14 ± 52.49	3324.87 ± 52.46
Intel/gemma-4-31B-it-int4-AutoRound	tg32 @ d8192	10.73 ± 0.16	11.33 ± 0.47
Intel/gemma-4-31B-it-int4-AutoRound	ctx_pp @ d32768	532.78 ± 2.76		66990.82 ± 388.63	66985.35 ± 388.63	66991.60 ± 388.71
Intel/gemma-4-31B-it-int4-AutoRound	ctx_tg @ d32768	9.66 ± 0.07	10.33 ± 0.47
Intel/gemma-4-31B-it-int4-AutoRound	pp2048 @ d32768	369.52 ± 1.01		5547.85 ± 15.13	5542.38 ± 15.13	5548.04 ± 15.14
Intel/gemma-4-31B-it-int4-AutoRound	tg32 @ d32768	9.54 ± 0.02	10.00 ± 0.00

llama-benchy (0.3.6.dev12+g5e7b509cb)
date: 2026-04-03 16:55:50 | latency mode: api

[TheAwakenOne]

I vibe coded a simple Chainlit chat + image + video UI in Pinokio to test Gemma-4, I figured everyone post stats I want to show you how it runs, the first section of the video is regular speed the second pass i sped it up because as you can tell its not the fastest, I think I have room to tweak it a bit and get better results maybe idk.. but I hope this helps, oh FYI it’s using 122-124GB out of the 128GB but at least it;s not crashing!

Music: ACE-Step via Pinokio

Running Gemma-4-31B-IT-NVFP4 on DGX Spark with vLLM 0.18.2rc1

• Model: nvidia/Gemma-4-31B-IT-NVFP4 (NVFP4 via ModelOpt)

• Quantization: modelopt

• Max model length: 65536

• GPU memory utilization: 0.85

• KV cache dtype: fp8

• Enabled: chunked prefill, prefix caching, reasoning_parser=gemma4, tool_call_parser=gemma4, auto tool choice

• Max num seqs: 4

• Max num batched tokens: 8192

• Attention backend: TRITON_ATTN (forced by heterogeneous heads)

• Host: 127.0.0.1:8000

Using for a desktop multimodal agent with screen, PDF, and video understanding.

Benchmark - first run: 2.5 tok/s · 341 out · 546 in · TTFT 2.23s · 137.3s total

[arctic.gus]

2.5 t/s seems very slow, i get 12t/s on mine with the same model.

[TheAwakenOne]

I created the identical app using Gemma-4-26B-A4B-IT and it’s much faster, though someone posted on reddit that they were getting 45-60 tok/s. but even with 10-12 tok/s running it on an application is not bad, if I can get it to 20 tok/s this is usable in applications:

Used this as a guide, need more tweaking to get faster.

Gemma 4 26B-A4B MoE running at 45-60 tok/s on DGX Spark — here’s how : r/LocalLLaMA

Running Gemma-4-26B-A4B-it with vLLM

• Model: google/gemma-4-26B-A4B-it (Mixture-of-Experts)

• Architecture: 25.2B total parameters / 3.8B active per token (128 experts, 8 active + 1 shared expert)

• Quantization: None (bf16 weights)

• Max model length: 32768 tokens (native support up to 256K)

• GPU memory utilization: 0.70

• KV cache dtype: fp8

• Enabled: chunked prefill, prefix caching, reasoning_parser=gemma4, tool_call_parser=gemma4, auto tool choice

• Max num seqs: 4

• Max num batched tokens: 8192

• Enforce eager: True (for stability on GB10)

• Attention backend: TRITON_ATTN (forced by heterogeneous heads

• Speed (tok/s): 10-12

[eugr]

Or you can use our community Docker at GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub
And then run this recipe:

./run-recipe.sh gemma4-26b-a4b --solo

It will run the same model at ~40 t/s on a single Spark.

[eugene.karasev]

Hi, @eugr , thank you for the great job! But it seems like latest build has some flaws. Just pulled and rebuild the TF5 image (git pull && ./build-and-copy.sh -t vllm-node-tf5 --tf5).

~/spark-vllm-docker$ ./run-recipe.sh gemma4-26b-a4b --solo --setup
Recipe: Gemma4-26B-A4B
  vLLM serving Gemma4-26B-A4B

Container 'vllm-node-tf5' already exists locally.

Model 'google/gemma-4-26B-A4B-it' already exists in cache.

=== Launching ===
Container: vllm-node-tf5
Mods: mods/fix-gemma4-tool-parser
Mode: Solo

Using launch script: /tmp/tmpnyovnw7e.sh
Solo mode enabled. Skipping node detection.
Head Node: 127.0.0.1
Worker Nodes:
Container Name: vllm_node
Image Name: vllm-node-tf5
Action: exec
Starting Head Node on 127.0.0.1...
86be2e1691f73eb1e6e5b99593a41da4d77ceaf2eca41836117222716b1009a8
Applying modifications to cluster nodes...
Applying mod 'fix-gemma4-tool-parser' to 127.0.0.1...
  Copying directory content to container...
Successfully copied 2.56kB to vllm_node:/workspace/mods/fix-gemma4-tool-parser/
  Running patch script on 127.0.0.1...
Applying PR #38909
- PR #38909 applied successfully
Copying launch script to head node...
Successfully copied 2.05kB to vllm_node:/workspace/exec-script.sh
Executing command: /workspace/exec-script.sh
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 4, in <module>
    from vllm.entrypoints.cli.main import main
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/__init__.py", line 3, in <module>
    from vllm.entrypoints.cli.benchmark.latency import BenchmarkLatencySubcommand
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/latency.py", line 5, in <module>
    from vllm.benchmarks.latency import add_cli_args, main
  File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/latency.py", line 15, in <module>
    from vllm.engine.arg_utils import EngineArgs
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 35, in <module>
    from vllm.config import (
  File "/usr/local/lib/python3.12/dist-packages/vllm/config/__init__.py", line 6, in <module>
    from vllm.config.compilation import (
  File "/usr/local/lib/python3.12/dist-packages/vllm/config/compilation.py", line 22, in <module>
    from vllm.platforms import current_platform
  File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/__init__.py", line 279, in __getattr__
    _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils/import_utils.py", line 111, in resolve_obj_by_qualname
    module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py", line 21, in <module>
    import vllm._C  # noqa
    ^^^^^^^^^^^^^^
ImportError: /usr/local/lib/python3.12/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN2at4cuda24getCurrentCUDABlasHandleEv

Stopping cluster...
Stopping head node (127.0.0.1)...
Cluster stopped.

The issue “vllm images can’t be built anymore” is already opened.

vLLM rebuild seems like workaround:

~/spark-vllm-docker$ ./build-and-copy.sh --rebuild-vllm -t vllm-node-tf5 --tf5

After vLLM rebuild, it runs successfully, but patch application was failed:

Applying mod 'fix-gemma4-tool-parser' to 127.0.0.1...
  Copying directory content to container...
Successfully copied 2.56kB to vllm_node:/workspace/mods/fix-gemma4-tool-parser/
  Running patch script on 127.0.0.1...
Applying PR #38909
error: patch failed: vllm/tool_parsers/gemma4_tool_parser.py:436
error: vllm/tool_parsers/gemma4_tool_parser.py: patch does not apply
- PR #38909 can't be applied, skipping

[eugr]

Yeah, I was traveling and a few things broke in the build pipeline. They have been fixed, and I’ll look into gemma 4 tool parser patch issue.