Host driver: 580.173.02 (installed via DGX Dashboard updates)
Problem: After a DGX Dashboard update moved the driver 580.159.03 → 580.173.02, the container now fails its compatibility check at startup:
“ERROR: This container was built for NVIDIA Driver Release 595.58 or later, but version 580.173.02 was detected and compatibility mode is UNAVAILABLE”.
On 580.159.03 the same container started with “CUDA Forward Compatibility mode ENABLED” and ran fine.
The container still falls back and serves requests, but not through the intended forward-compatibility path. I can’t move to higher version because of known incompatibilities or instabilities.
Question: What is the supported driver + vLLM-container combination for DGX Spark today? Will a >=595.58 driver be delivered via DGX Dashboard, and if so when? Until then, will the NGC vLLM images remain runnable on the 580.x branch you recommend for GB10?
Thanks! Tried both commands, but still the same: compatibility mode is UNAVAILABLE on 580.173.02.
Note: I launch with --gpus all (not CDI), and the container already ran fine, so regenerating the CDI spec doesn’t change the check. This still looks like a driver version gap (580.x < 595.58) rather than a container-toolkit issue.
Thanks Mark, useful lead. Confirmed on my 580.173.02: the CUDA 13.1 container (26.02) does get “Forward Compatibility mode ENABLED (CUDA 13.1 / 590.48.01)”, so the driver’s forward-compat isn’t broken, it’s just capped at CUDA 13.1.
The catch: 26.02 (vLLM 0.15.1 / older transformers) doesn’t recognize my model (‘unsloth/Qwen3.6-27B-NVFP4’). It fails with “model type ‘qwen3_5’ … Transformers does not recognize this architecture”. And the tags that do support Qwen3.6 (26.03+, incl. my 26.04) are CUDA 13.2, needing driver ≥595.58, above my forward-compat ceiling.
So, as far as I can see, on 580.173.02 no NGC tag both forward-compats and runs Qwen3.6: model support (26.03+/CUDA 13.2) and forward-compat (caps at CUDA 13.1) sit on opposite sides of that line.
Real fix seems to be either a ≥595.58 driver (via official DGX Dashboard), or a CUDA-13.1 vLLM build new enough to include qwen3_5 (the model_type of Qwen3.6, as reported in the error I’ve got). Appreciate it!
This is a known compatibility gap: the DGX Dashboard currently ships driver 580.173.02 while vLLM 26.04-py3 requires ≥595.58 for full forward-compatibility mode.
Could you confirm: (1) exact container image tag/digest, (2) output of nvidia-smi and cat /proc/driver/nvidia/version?
We’re tracking the driver upgrade path internally. In the meantime, vLLM 25.05.1-py3 should work on driver 580.
Tag: nvcr.io/nvidia/vllm:26.04-py3
docker inspect … RepoDigests returns empty on my host (no stored registry digest); local image ID (.Id):
sha256:922ce0c54006435b986034d74f81347b33da0d9fa1eb38bcbab950d04e126bb1
Startup banner: NVIDIA Release 26.04 (build 299333414), vLLM 0.19.0+6bc3197f
nvidia-smi:
Driver Version: 580.173.02
CUDA Version: 13.0
GPU: NVIDIA GB10
/proc/driver/nvidia/version:
NVRM version: NVIDIA UNIX Open Kernel Module for aarch64 580.173.02 Release Build (dvs-builder@U22-A24-5-4) Tue Jun 23 08:34:19 UTC 2026
GCC version: gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)
On the 25.05.1-py3 suggestion, it isn’t pullable (“manifest unknown”).
The earliest vLLM tags I can find are ~25.10/25.12 (vLLM 0.11.1) and 26.01 (vLLM 0.13).
I don’t see a 25.05. Could you confirm the exact tag/digest?
One more thing that narrows the options: my checkpoint unsloth/Qwen3.6-27B-NVFP4 uses the qwen3_5 architecture and needs vLLM ≥0.19. I already tested 26.02-py3 (vLLM 0.15.1); it forward-compats fine on driver 580 (Forward Compatibility mode ENABLED, CUDA 13.1) but fails to load the model (model type ‘qwen3_5’).
Transformers does not recognize that architecture. So any tag old enough to forward-compat on 580 (CUDA 13.1) is vLLM ≤0.15 and predates Qwen3.6, while the tags that support it (26.03+, incl. my 26.04) are CUDA 13.2 and need ≥595.58. On 580 there’s no NGC tag that both forward-compats and runs qwen3_5, so the fix needs either the ≥595.58 driver you’re tracking, or qwen3_5 support backported into a CUDA-13.1 image.
Hi @fmarcano — thank you for the detailed follow-up and for catching the 25.05.1-py3 error in my previous reply. That tag does not exist on NGC; I apologize for the confusion.
Your analysis in post 8 is exactly right. On driver 580.173.02, the forward-compat layer caps at CUDA 13.1, which means the highest NGC vLLM tag that forward-compats cleanly is 26.02 (vLLM 0.15.1). But qwen3_5 support requires vLLM ≥0.19, which first ships in 26.03+. Those containers are built for CUDA 13.2 and require driver ≥595.58 — above your forward-compat ceiling. There is no NGC tag that currently bridges both requirements on 580.x.
What’s coming: We are tracking a driver upgrade path that would bring ≥595.58 to DGX Dashboard. I don’t have a confirmed delivery date to share yet, but will update this thread when I do.
Workarounds until then:
eugr’s pre-built vLLM images for DGX Spark (most practical right now) — @eugr maintains community-built vLLM Docker images specifically tuned for GB10. They include NVIDIA_DISABLE_REQUIRE: 1, which bypasses the container’s startup driver-version check, allowing a newer vLLM build to run on driver 580.x. Community members have validated this path for Qwen3.6-27B NVFP4 inference on DGX Spark — see the Slow Qwen3.6 27B NVFP4 recipe thread and the main eugr vLLM images thread for setup instructions via Sparkrun. Note that bypassing the driver check is not officially supported: if the container issues CUDA 13.2 calls that require kernel features only in 595.58+, you could see runtime failures rather than a clean startup error. Worth trying first given the community validation.
26.07-py3 — the July NGC container ships vLLM 0.23+ with qwen3_5 support. It is also built for CUDA 13.2, so it likely carries the same ≥595.58 forward-compat requirement as 26.04 — but it’s worth pulling and testing, as the compat floor is not always identical between releases.
Host-native vLLM from source — building vLLM ≥0.19 directly on the host against your existing CUDA 13.0 toolkit sidesteps the container forward-compat check entirely. The custom vLLM build thread (Qwen3.5-35B on GB10, sustained 50 tok/s) documents a validated approach for this on DGX Spark.
I’ll update this thread once I have a confirmed timeline on the driver upgrade. Thanks again for the thorough diagnostics — this is a genuinely thorny compat gap and your writeup will help others who hit the same wall.
Thanks Neill, much appreciated. Glad the writeup helps, and good to know the ≥595.58 Dashboard path is being tracked.
I’ll stay on 26.04-py3 (native fallback, ~17–20 tok/s) and wait for the driver rather than bypass the check, since I want to keep this box on the officially-supported path as possible.
One note for anyone landing here via the tool-calling route: 26.07-py3 does ship qwen3_5 support, but it currently returns HTTP 500 on any tool_choice request (cannot import name ‘normalize_tool_choice’ from ‘xgrammar’, shipped xgrammar 0.2.0 vs the ≥0.2.4 its vLLM build calls; tracked in a separate thread: Nvcr.io/nvidia/vllm:26.07-py3 — tool-calling requests 500 due to xgrammar/transformers version).
So even where it starts, it isn’t a clean workaround for agentic tool-calling use until that’s fixed.
Thanks again. I’ll watch the thread for the driver timeline.