Run VLLM in Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
61

Thank you so much @eugr , you’ve been a great help!

I wanted to share my Dockerfile with everyone; it works perfectly. Have fun! :)

Dockerfile

FROM nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04

# Install essentials
RUN apt-get update && apt-get install -y \
    python3.12 python3.12-venv python3-pip git wget patch \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Create virtual env
RUN python3.12 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Upgrade pip
RUN pip install --upgrade pip

# Install PyTorch + CUDA
RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

# Install pre-release deps
RUN pip install xgrammar triton flashinfer-python --pre

# Clone vLLM
RUN git clone https://github.com/vllm-project/vllm.git
WORKDIR /app/vllm

RUN python3 use_existing_torch.py

RUN pip install -r requirements/build.txt

# Apply patch
COPY vllm_patch.diff .
RUN patch -p1 < vllm_patch.diff

RUN apt-get update && apt-get install -y \
    cmake \
    build-essential \
    ninja-build \
    && rm -rf /var/lib/apt/lists/*

# Set essential environment variables
ENV TORCH_CUDA_ARCH_LIST=12.0f
ENV TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
ENV TIKTOKEN_ENCODINGS_BASE=/app/tiktoken_encodings

# Install vLLM with local build
RUN pip install --no-build-isolation -e . -v --pre

# Download tiktoken encodings
WORKDIR /app
RUN mkdir -p tiktoken_encodings && \
    wget -O tiktoken_encodings/o200k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken" && \
    wget -O tiktoken_encodings/cl100k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken"


WORKDIR /app/vllm

RUN pip install vllm[audio]

# Expose port
EXPOSE 8888

ENTRYPOINT []

vllm_patch.diff (thanks again @eugr for this patch )

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 7cb94f919..f860e533e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -594,9 +594,9 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
 
   # FP4 Archs and flags
   if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
-    cuda_archs_loose_intersection(FP4_ARCHS "10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
+    cuda_archs_loose_intersection(FP4_ARCHS "10.0f" "${CUDA_ARCHS}")
   else()
-    cuda_archs_loose_intersection(FP4_ARCHS "10.0a;10.1a;12.0a;12.1a" "${CUDA_ARCHS}")
+    cuda_archs_loose_intersection(FP4_ARCHS "10.0a;10.1a" "${CUDA_ARCHS}")
   endif()
   if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND FP4_ARCHS)
     set(SRCS
@@ -668,7 +668,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
   endif()
 
   if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
-    cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0f;11.0f" "${CUDA_ARCHS}")
+    cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0f" "${CUDA_ARCHS}")
   else()
     cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0a" "${CUDA_ARCHS}")
   endif()
@@ -716,9 +716,9 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
   endif()
 
   if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
-    cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
+    cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0f" "${CUDA_ARCHS}")
   else()
-    cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0a;10.1a;10.3a;12.0a;12.1a" "${CUDA_ARCHS}")
+    cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0a;10.1a;10.3a" "${CUDA_ARCHS}")
   endif()
   if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND SCALED_MM_ARCHS)
     set(SRCS "csrc/quantization/w8a8/cutlass/moe/blockwise_scaled_group_mm_sm100.cu")
62

Ok thx.Aactually I use Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 (based on your Dockerfile).

@eugr Model compatibility and conversion are complex topics; are there any tutorials (besides Nvidia playbooks) that you recommend for making progress on these subjects?

I’m sharing my Docker Compose setup in case it helps someone.

version: '3.8'

services:
  vllm-server:
    container_name: qwen-vllm-custom
    build:
      context: .
      dockerfile: Dockerfile
    image: vllm-custom:25.09
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8888:8000"
    volumes:
      - /home/misteroo/.cache/huggingface:/root/.cache/huggingface
      - /home/misteroo/.cache/vllm:/root/.cache/vllm
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    ulimits:
      memlock: -1
      stack: 67108864
    command: >
      vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --max-model-len 128000 --host 0.0.0.0 --gpu-memory-utilization 0.8
    restart: always

  # 2. Service d'Interface Utilisateur (Open-WebUI)
  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:main
    depends_on:
      - vllm-server
    ports:
      - "3000:8080"
    volumes:
      - open-webui-data:/app/backend/data
    environment:
      OPENAI_API_KEY: "EMPTY"
      OPENAI_API_BASE_URL: http://vllm-server:8000/v1
    restart: always

  embeddings-server:
    container_name: llama-3.2-nemoretriever-300m-embed-v2
    image: nvcr.io/nim/nvidia/llama-3.2-nemoretriever-300m-embed-v2:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    shm_size: "16gb"
    environment:
      - NGC_API_KEY=${NGC_API_KEY}
    user: "${UID:-1000}:${GID:-1000}"
    volumes:
      - ~/.cache/nim:/opt/nim/.cache
    ports:
      - "8000:8000"
    restart: unless-stopped

volumes:
  open-webui-data:
63

Hi everyone,
Has anyone successfully managed to run gpt-oss-20b on vLLM with tool calling working correctly?
I’m using the image nvcr.io/nvidia/vllm:25.09-py3, but I keep getting this error:

openai.BadRequestError: Error code: 400 - {‘error’: {‘message’: “‘NoneType’ object is not iterable None”, ‘type’: ‘BadRequestError’, ‘param’: None, ‘code’: 400}}

64

Just use suitable model from Huggingface. Qwen usually publishes their FP8 versions themselves, for other models you can click on Quantizations at the model page and look up suitable quant.

VLLM is usually the first major inference engine to get new models support, and the model page usually includes instructions on how to run it there. Just need to compile fresh vllm out of main branch once a new supported model comes along.

65

i’m able to use Dockerfile and patch @vince_du_66 provide to run GPT OSS 20b without tool calling issue

66

There are some fixes from flashinfer future release for spark.
We are working hard, thanks for all your feedback. You can test it on flash infer released daily nightly builds, starting 11/14/2025.

I recommend also jump to triton 3.5.1 and pytorch 2.9.1 released yesterday with cu130.

67

@giacomobellini97 This works. Note that vllm:25.10-py3. I didn’t check, but by the time I finish writing this, it might 25.11-py3. Who knows.

$ docker run -it --gpus all -p 8000:8000  --ipc=host \
       -v ${HF_HOME}:/root/.cache/huggingface nvcr.io/nvidia/vllm:25.10-py3 \
       vllm serve "openai/gpt-oss-20b" --host 0.0.0.0 --port 8000 \
       --gpu-memory-utilization 0.5 

$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "openai/gpt-oss-20b",
     "messages": [{"role": "user", "content": "What is Life?"}],
     "max_tokens": 500
 }'| jq
68

@johnny_nv I tried to get that combination working without building it, but haven’t been able to run. It’s always either CUDA not found or flashinfer error. Dockers for vLLM, llama.cpp, SGLang, TensortRT-LLM works great though. But yea, I wish I can use them without docker “reliably”. Only if someone can provide a Jupyter notebook or a shell script that REALLY works every time, s/he shall be hero!.. till the next update to one of library that break it.

69
70

FROM nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04

# Install essentials

RUN apt-get update && apt-get install -y \

python3.12 python3.12-venv python3-pip git wget patch \\

&& rm -rf /var/lib/apt/lists/\*

# Set working directory

WORKDIR /app

# Create virtual env

RUN python3.12 -m venv /opt/venv

ENV PATH=“/opt/venv/bin:$PATH”

# Upgrade pip

RUN pip install --upgrade pip

# Install PyTorch + CUDA

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

# Install pre-release deps

RUN pip install xgrammar triton

RUN pip install -U --pre flashinfer-python --index-url Index of nightly --no-deps

RUN pip install flashinfer-python

RUN pip install -U --pre flashinfer-cubin --index-url Index of nightly

# JIT cache package (replace cu129 with your CUDA version: cu128, cu129, or cu130)

RUN pip install -U --pre flashinfer-jit-cache --index-url Index of cu130

# Clone vLLM

RUN git clone GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs

WORKDIR /app/vllm

RUN git fetch origin pull/26844/head:pr-26844

RUN git -c user.name=“CI Bot” -c user.email="ci@example.com" merge --no-ff --no-edit pr-26844

RUN python3 use_existing_torch.py

RUN sed -i “/flashinfer/d” requirements/cuda.txt

RUN pip install -r requirements/build.txt

RUN apt-get update && apt-get install -y \

cmake \\

build-essential \\

ninja-build \\

&& rm -rf /var/lib/apt/lists/\*

# Set essential environment variables

ENV TORCH_CUDA_ARCH_LIST=12.1a

ENV TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas

ENV TIKTOKEN_ENCODINGS_BASE=/app/tiktoken_encodings

# Install vLLM with local build

RUN pip install --no-build-isolation -e . -v --pre

RUN pip install --no-build-isolation -e .[audio] -v --pre

# Download tiktoken encodings

WORKDIR /app

RUN mkdir -p tiktoken_encodings && \

wget -O tiktoken_encodings/o200k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken" && \\

wget -O tiktoken_encodings/cl100k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken"

WORKDIR /app/vllm

# Expose port

EXPOSE 8888

ENTRYPOINT

This builds successfully for me, posting here if anyone should need it. 11/14 flashinfer nightly included.

72

There is a new patch for CMakeLists.txt - now tracked in this PR.
To apply, you can run:

curl -L https://patch-diff.githubusercontent.com/raw/vllm-project/vllm/pull/28938.diff | git apply

Hopefully this gets merged quickly :)

73

Was merged. 🥳 And they just released v0.11.1, but without it as it seems.

74

Yeah, 0.11.1 was released a few hours earlier :)

75

they generate nightly daily builds, so you can try it

76

I successfully ran Mistral-7b using the instructions provided in this thread. Thanks !
Here is my installation script:

sudo apt install cmake build-essential ninja-build python3-dev

uv venv --python 3.12
source .venv/bin/activate
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130
uv pip install xgrammar triton
uv pip install flashinfer-python --prerelease=allow --index-url https://flashinfer.ai/whl/nightly/ --no-deps
uv pip install flashinfer-cubin --index-url https://flashinfer.ai/whl/nightly/
uv pip install flashinfer-jit-cache --prerelease=allow --index-url https://flashinfer.ai/whl/nightly/cu130

git clone https://github.com/vllm-project/vllm.git
cd vllm
python use_existing_torch.py
sed -i “/flashinfer/d” requirements/cuda.txt
uv pip install -r requirements/build.txt

export TORCH_CUDA_ARCH_LIST=12.1a
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
export TIKTOKEN_ENCODINGS_BASE=$PWD/tiktoken_encodings
uv pip install --no-build-isolation -e . -v --pre
77

One correction, for CUDA 13, Spark arch should be 12.0f:

export TORCH_CUDA_ARCH_LIST=12.0f
78

no –> NVIDIA Blackwell and NVIDIA CUDA 12.9 Introduce Family-Specific Architecture Features | NVIDIA Technical Blog

No suffix: Your PTX or cubin compatibility is the same as it always has been.

f suffix: Whether you stop at PTX or generate cubin from that code, that code is compatible to run on GPU devices with the same major compute capability version and with an equal or higher minor compute capability version. 12.0f you are activating features for 12.0 family, 12.0 and 12.1. But it can be some features(tensor cores that can be specific in chip that with this flag, you are not activating).

a suffix: The code only runs on GPUs of that specific CC and no others. With a you obtain all specific features from the chip.

So it depends in the project and the kernels, but if you are building only for one device, and you are planning to use tensor cores, I recommend to use a

79

Based on my understanding, it changed with CUDA 13, but I may be wrong.

In any case, if you specify 12.1a, a lot of Blackwell features will not get compiled into VLLM with CUDA 13, because CMakeLists.txt is peppered with constructs like this:

if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
    cuda_archs_loose_intersection(SCALED_MM_ARCHS "12.0f" "${CUDA_ARCHS}")
  else()
    cuda_archs_loose_intersection(SCALED_MM_ARCHS "12.0a" "${CUDA_ARCHS}")
  endif()

# The nvfp4_scaled_mm_sm120 kernels for Geforce Blackwell SM120 require
  # CUDA 12.8 or later
  if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
    cuda_archs_loose_intersection(FP4_ARCHS "12.0f" "${CUDA_ARCHS}")
  else()
    cuda_archs_loose_intersection(FP4_ARCHS "12.0a" "${CUDA_ARCHS}")
  endif()

For 12.1 specifically:

# CUTLASS MLA Archs and flags
  if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
    cuda_archs_loose_intersection(MLA_ARCHS "10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
  else()
    cuda_archs_loose_intersection(MLA_ARCHS "10.0a;10.1a;10.3a;12.0a;12.1a" "${CUDA_ARCHS}")
  endif()
80

No, this is the work from the compiler.
Putting 12.1 or 12.1a or 12.1f is the same for cuda kernels. For tensor core kernels, it is specifiyng which features use.
Anyways vllm does a preprocessing that it take only the numbers without letters to check which arch build.

81

I see upstream released a new wheel with one of the fixes for DGX Spark, should we expect the aarch64 wheel to be compatible with DGX Spark and run accelerated workloads soon?