Docker container image for recent vLLM release that enables GGUF loading

Accelerated Computing NGC GPU Cloud Docker and NVIDIA Docker
1

As explained in this post, the community would appreciate a more recent NGC vLLM image. The current image (nvcr.io/nvidia/vllm:25.09-py3) does not support GGUF loading.

The workaround is to compile against the 25.09 base (as shown in the earlier post).

Specifically, I was trying to load gemma3-4b-it-q4_0, but that is blocked by another issue:

Please correct/refine if you see any errors with this Dockerfile and invocation.

Dockerfile

# Start with the official NVIDIA NGC vLLM image.
# This provides the necessary CUDA, PyTorch, and Python environment.
FROM nvcr.io/nvidia/vllm:25.09-py3

WORKDIR /workspace 

RUN git clone https://github.com/vllm-project/vllm.git && \
  cd vllm && \
  python3 use_existing_torch.py && \
  pip3 install -r requirements/build.txt && \
  MAX_JOBS=12 pip3 install -e . --no-build-isolation

# simple entrypoint for interactive use
CMD ["/bin/bash"]

Invocation (from the container)

# vLLM API server version 0.11.1rc2.dev165+gd31f7844f.d20251019
CHECKPOINT=/root/.cache/huggingface/gemma-3-4b-it-q4_0.gguf
MODEL=google/gemma-3-4b-it

vllm serve ${CHECKPOINT} --tokenizer ${MODEL}
2

We recently released a new vLLM container, 25.10-py3. Please try this one and let us know if it works.

3

Appreciate the help! Thank you. Unfortunately, the quantization issues won’t be solved until vLLM 0.11.x is included, so I’ll just add some instructions to my post and acknowledge the updated container image.

4

It’s great to see an updated NGC vLLM container available, which includes vLLM v0.10.2! Thank you!

If you’d like to use a more recent version of vLLM (like v0.11.x so you can serve quantization-aware Gemma3 checkpoints), you can actually build an image right on the DGX Spark in a couple of hours.

You can start with this Dockerfile (this uses vLLM 0.11.1rc4)

# Start with the official NVIDIA NGC vLLM image
# This provides the necessary CUDA, PyTorch, and Python environment.
FROM nvcr.io/nvidia/vllm:25.10-py3

WORKDIR /workspace 

# Build vllm with the installed CUDA and PyTorch
# https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#use-an-existing-pytorch-installation
RUN git clone --branch v0.11.1rc4 https://github.com/vllm-project/vllm.git && \
  cd vllm && \
  python3 use_existing_torch.py && \
  pip3 install -r requirements/build.txt && \
  MAX_JOBS=12 pip3 install -e . --no-build-isolation

# simple entrypoint for interactive use
CMD ["/bin/bash"]

Next, you can build the Docker container image

docker build \
  --build-arg max_jobs=120 \
  --build-arg nvcc_threads=20 \
  --build-arg torch_cuda_arch_list="13.0" \
  --build-arg CUDA_VERSION="13.0.1" \
  --platform "linux/arm64" \
  --file dgx-spark-cu13.Dockerfile \
  --tag vllm-spark-cu13 .

You can start the container using something like this

HF_TOKEN=...  # your HuggingFace auth token here
DOCKER_IMAGE="vllm-spark-cu13:latest"

docker run --gpus all --rm -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface  \
  -p 8000:8000 \
  --env "HF_TOKEN=${HF_TOKEN}" \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  ${DOCKER_IMAGE}

In the container, you can serve the unsloth Gemma (quantization-aware) using something like this (presuming you’ve downloaded the checkpoint from Hugging Face already):

export MODEL="unsloth/gemma-3-4b-it"
export MODEL_NAME="unsloth_gemma-3-4b-it-q4_0"
export CHECKPOINT="/root/.cache/huggingface/unsloth_gemma-3-4b-it-Q4_0.gguf"

vllm serve ${CHECKPOINT} \
  --tokenizer ${MODEL} \
  --served-model ${MODEL_NAME} \
  --gpu-memory-utilization 0.75
5

In the latest vLLM container, 26.03, GGUF on Blackwell works as expected. You would still need to add --dtype float32 but other dtypes are now automatically rejected by vLLM anyways.