New NGC vLLM container image (vllm:26.01-py3)

dlewis.io · January 31, 2026, 3:15pm

Looks like the release notes haven’t been updated, yet. Running it now with on my 2x DGX Sparks, and it seems to work well.

https://docs.nvidia.com/deeplearning/frameworks/vllm-release-notes/rel-26-01.html#rel-26-01

Here are the changes that I pulled from 26.01 and 25.12.post1. vLLM version bump to 0.13.0 appears to be the biggest change.

Core Components:

vLLM: 0.12.0 → 0.13.0 (minor version bump)
CUDA: 13.1.0.036 → 13.1.1.006
CUDA Driver: 590.44.01 → 590.48.01
cuDNN: 9.17.0.29 → 9.17.1.4
cuDNN Frontend: 1.16.0 → 1.17.0
NCCL: 2.28.9 → 2.29.stable.20260109
Transformer Engine: 2.10 → 2.11
DALI: 1.52.0 → 1.53.0
Model Optimizer: 0.39.0 → 0.40.0

Python Libraries:

FlashInfer: 0.6.0rc2 → 0.6.0 (release candidate → stable)
xformers: 0.0.33+nv25.12.post1 → 0.0.33+nv26.1
FastAPI: 0.127.0 → 0.128.0
OpenAI SDK: 2.14.0 → 2.15.0
websockets: 15.0.1 → 16.0
scipy: 1.16.3 → 1.17.0
tokenizers: 0.22.1 → 0.22.2
pillow: 12.0.0 → 12.1.0
typer: 0.20.1 → 0.21.1

dlewis.io · January 31, 2026, 6:57pm

FYI –async-scheduling still seems to be flaky with vLLM 0.13.0. Getting CUDA graph crashes with it turned on. Disabling it has fixed it.

christopher_owen · January 31, 2026, 7:25pm

This should depend on the model being used, no?

NVES · January 31, 2026, 7:27pm

would love to learn more about this and pass this on to eng. any context/repro steps to demonstrate what you are seeing?

trystan1 · January 31, 2026, 8:43pm

Don’t know if this is what dlewis is seeing, but with long context prompts I’m getting 500 errors from vllm with this being shown in dmesg (using both newest vllm ngc containers)

[ 3886.179778] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 0): Illegal Instruction Parameter

[ 3886.179787] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 1, SM 0): Multiple Warp Errors

[ 3886.179792] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x516730=0x2000b 0x516734=0x24 0x516728=0x1c81fb60 0x51672c=0x1174

[ 3886.179796] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 1): Illegal Instruction Parameter

[ 3886.179965] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 1, SM 1): Multiple Warp Errors

[ 3886.180352] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5167b0=0x1000b 0x5167b4=0x24 0x5167a8=0x1c81fb60 0x5167ac=0x1174

[ 3886.180357] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 0): Illegal Instruction Parameter

[ 3886.180451] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 3, SM 0): Multiple Warp Errors

[ 3886.180531] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x518730=0x1000b 0x518734=0x24 0x518728=0x1c81fb60 0x51872c=0x1174

[ 3886.180790] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 1): Illegal Instruction Parameter

[ 3886.180889] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 3, SM 1): Multiple Warp Errors

[ 3886.180979] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5187b0=0x1000b 0x5187b4=0x24 0x5187a8=0x1c81fb60 0x5187ac=0x1174

[ 3886.181289] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 5, SM 0): Illegal Instruction Parameter

[ 3886.181388] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 5, SM 0): Multiple Warp Errors

[ 3886.181473] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x51a730=0x2000b 0x51a734=0x24 0x51a728=0x1c81fb60 0x51a72c=0x1174

[ 3886.181711] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 5, SM 1): Illegal Instruction Parameter

[ 3886.181814] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 5, SM 1): Multiple Warp Errors

[ 3886.181891] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x51a7b0=0x2000b 0x51a7b4=0x24 0x51a7a8=0x1c81fb60 0x51a7ac=0x1174

[ 3886.182224] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 0): Illegal Instruction Parameter

[ 3886.182319] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 1, SM 0): Multiple Warp Errors

[ 3886.182397] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x526730=0x3000b 0x526734=0x24 0x526728=0x1c81fb60 0x52672c=0x1174

[ 3886.182669] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 1): Illegal Instruction Parameter

[ 3886.182767] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 1, SM 1): Multiple Warp Errors

[ 3886.182845] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5267b0=0x3000b 0x5267b4=0x24 0x5267a8=0x1c81fb60 0x5267ac=0x1174

[ 3886.183149] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 1): Illegal Instruction Parameter

[ 3886.183246] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 2, SM 1): Multiple Warp Errors

[ 3886.183332] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5277b0=0x3000b 0x5277b4=0x24 0x5277a8=0x1c81fb60 0x5277ac=0x1174

[ 3886.183625] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 0): Illegal Instruction Parameter

[ 3886.183719] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x528730=0x2000b 0x528734=0x20 0x528728=0x1c81fb60 0x52872c=0x1174

[ 3886.183938] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 1): Illegal Instruction Parameter

[ 3886.184042] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5287b0=0x2000b 0x5287b4=0x20 0x5287a8=0x1c81fb60 0x5287ac=0x1174

[ 3886.184333] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 0): Illegal Instruction Parameter

[ 3886.184427] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x52a730=0xb 0x52a734=0x20 0x52a728=0x1c81fb60 0x52a72c=0x1174

[ 3886.184677] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 1): Illegal Instruction Parameter

[ 3886.184768] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x52a7b0=0x3000b 0x52a7b4=0x20 0x52a7a8=0x1c81fb60 0x52a7ac=0x1174

[ 3886.185096] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 1, SM 0): Illegal Instruction Parameter

[ 3886.185189] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x536730=0x1000b 0x536734=0x20 0x536728=0x1c81fb60 0x53672c=0x1174

[ 3886.185401] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 1, SM 1): Illegal Instruction Parameter

[ 3886.185498] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5367b0=0x3000b 0x5367b4=0x20 0x5367a8=0x1c81fb60 0x5367ac=0x1174

[ 3886.185791] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 0): Illegal Instruction Parameter

[ 3886.185884] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x537730=0x2000b 0x537734=0x20 0x537728=0x1c81fb60 0x53772c=0x1174

[ 3886.186156] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 3, SM 0): Illegal Instruction Parameter

[ 3886.186252] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x538730=0x3000b 0x538734=0x20 0x538728=0x1c81fb60 0x53872c=0x1174

[ 3886.186484] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 3, SM 1): Illegal Instruction Parameter

[ 3886.186585] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5387b0=0x1000b 0x5387b4=0x20 0x5387a8=0x1c81fb60 0x5387ac=0x1174

[ 3886.186890] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 5, SM 0): Illegal Instruction Parameter

[ 3886.186984] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x53a730=0x1000b 0x53a734=0x20 0x53a728=0x1c81fb60 0x53a72c=0x1174

[ 3886.187199] NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 5, SM 1): Illegal Instruction Parameter

[ 3886.187292] NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x53a7b0=0xb 0x53a7b4=0x20 0x53a7a8=0x1c81fb60 0x53a7ac=0x1174

[ 3886.190317] NVRM: Xid (PCI:000f:01:00): 43, pid=20919, name=VLLM::EngineCor, channel 0x00000002

I’ve also tried with the released 6.14 kernel, and the newer 6.17 kernel with 590 version modules, same results. This seems to be only with async scheduling enabled.

Update: happened with async-scheduling disabled as well, seems like this is a nemotron nano 3 nvfp4 version issue and not async scheduling problem.

solomun.marko · May 3, 2026, 1:59pm

I’ve came across of this in search for vllm images that could be optimized better compared to vllm/vllm-openai:latest

I am new to this, running WSL and RTX 6000 Pro and so far I’ve tested few models and Qwen3.6 27B FP8 runs at 70-120tok/s in generation. I wonder if NVFP4 version of it would give better results.

I am running this (below) docker compose but getting 20-25tok/s and would assume config is wrong or I am using wrong vllm image. I am not quiet sure how this nvidia’s vllm image shuld be ran in similar configuration.

The model used below is mmangkad/Qwen3.6-27B-NVFP4 · Hugging Face

I could not find Nvidia’s Qwen3.6 27B NVFP4 version (is there one?).

If someone could stear me in right direction I would appreciate it.

services:
  vllm-large:
    image: vllm/vllm-openai:latest
    container_name: vllm-large
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      - VLLM_API_KEY=some-secure-key-here
      - HF_HOME=/root/.cache/huggingface
    volumes:
      - /mnt/p/models/qwen:/models
      - /mnt/p/hf-cache:/root/.cache/huggingface

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

    ipc: host

    command: >
      --model /models/Qwen3.6-27B-NVFP4-unsloth
      --served-model-name qwen3.6-27b-nvfp4-unsloth
      --trust-remote-code
      --max-model-len 196608
      --max-num-seqs 256
      --max-num-batched-tokens 32768
      --gpu-memory-utilization 0.70
      --dtype bfloat16
      --kv-cache-dtype fp8
      --attention-backend flashinfer
      --enable-prefix-caching
      --no-scheduler-reserve-full-isl
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --enable-chunked-prefill
      --override-generation-config '{"max_new_tokens": 81920}'
      --default-chat-template-kwargs '{"enable_thinking": true}'
      --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
    
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s

jwarner · May 3, 2026, 6:14pm

First, use the community Docker.

Second, use PrismaQuant 5.5 bit for Qwen3.6-27B, with built in MTP set to 3 positions.

Really fantastic model, acceptable throughput for the quality, extremely high fidelity (I don’t love int4-Auroround but this one is basically equivalent to FP8 to me).

solomun.marko · May 3, 2026, 6:47pm

Could you explain what you mean by community Docker. And the PrismaQuant you men model rdtand/Qwen3.6-27B-PrismaQuant-5.5bit-vllm · Hugging Face

UPDATE: I’ve tried PrismaQuant and finally made it to work using vLLM version 0.20.2rc1.dev5+xxxxx The results are impressive, uplift of 20-45tok/s from FP8 version.

Accepted throughput: 70.99 tokens/s,
Drafted throughput: 80.09 tokens/s
Avg generation throughput: 91.6 tokens/s

I had to ran it directly in WSL with latest CUDA toolkit installed. It may need some tuning but this is where I am at right now.

vllm serve /mnt/p/models/rdtand/Qwen3.6-27B-PrismaQuant-5.5bit-vllm --served-model-name Qwen3.6-27B-PrismaQuant-5.5bit-vllm --trust-remote-code --max-model-len 120768 --gpu-memory-utilization 0.90 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

I’ve mainly tested Qwen/Qwen3.6-27B-FP8 · Hugging Face and getting 35-45tok/per (in LM Studio and vLLM) request. I am more focused on single-user and output per user.

Conclusion (so far) for single user LM Studio may be better way to go for ease of use and quick model switching. Performance is near identical.

Latest what I managed to pull off from NVFP4 variant is 50tok/s per request, and 25 paralel requests with 900tok/s in total on 500W power budget.

I do not need 10+ paralel requests, but 2-4 max. Ive disabled MTP since it was slowing generation by 4x.

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

services:
  vllm-large:
    image: nvcr.io/nvidia/vllm:26.04-py3
    container_name: vllm-large
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      - VLLM_API_KEY=some-secure-key-here
      - HF_HOME=/root/.cache/huggingface
    volumes:
      - /mnt/p/models/LLM_Sudio/models/qwen:/models
      - /mnt/p/hf-cache:/root/.cache/huggingface

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

    ipc: host

    command: >
      python3 -m vllm.entrypoints.openai.api_server
      --model /models/Qwen3.6-27B-NVFP4-unsloth
      --served-model-name qwen3.6-27b-nvfp4-unsloth
      --trust-remote-code
      --max-model-len 32768
      --max-num-seqs 256
      --max-num-batched-tokens 65536
      --gpu-memory-utilization 0.70
      --dtype bfloat16
      
      --attention-backend flashinfer
      --enable-prefix-caching
      --no-scheduler-reserve-full-isl
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --enable-chunked-prefill
      --override-generation-config '{"max_new_tokens": 81920}'
      --default-chat-template-kwargs '{"enable_thinking": true}'
      --tensor-parallel-size 1
      --tokenizer Qwen/Qwen3.5-27B
    
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s # Increased for large model load times

Topic		Replies	Views
vLLM containers DGX Spark / GB10	45	2918	July 22, 2026
Announcing new VLLM container & 3.5X increase in Gen AI Performance in just 5 weeks of Jetson AGX Thor Launch Jetson Thor jetson , llama-31-8b-instruct , llama , deepseek , nemotron	45	4275	December 14, 2025
New pre-built vLLM Docker Images for NVIDIA DGX Spark DGX Spark / GB10	73	10385	March 27, 2026
Nemotron-3-Nano-30B-A3B-NVFP4 ultra-efficient NVFP4 precision version of Nemotron 3 Nano DGX Spark / GB10 jetson , nemotron	84	3762	March 20, 2026
vLLM container 25.10-py3 fails to start Jetson Thor nvbugs , generative_ai	13	803	December 8, 2025
GLM-4.7-Flash-NVFP4 was just released, but for Transformers 5.0 + vLLM 0.14...? DGX Spark / GB10	89	5010	February 13, 2026
New bleeding-edge vLLM Docker Image: avarok/vllm-nvfp4-gb10-sm120 DGX Spark / GB10 Projects	32	3496	December 17, 2025
Docker container image for recent vLLM release that enables GGUF loading Docker and NVIDIA Docker	4	1301	April 14, 2026
Nvcr.io/nvidia/vllm:25.12-py3 Fails to Execute Qwen3-VL-8B Image Inference with Engine Error Jetson Thor generative_ai	10	711	January 7, 2026
Who wants to be the hero and help a total newbie! Got a spark and um, yeah DGX Spark / GB10 nemotron	7	774	April 3, 2026

New NGC vLLM container image (vllm:26.01-py3)

Related topics