Request for clarification: forward compatibility behavior on driver 550.54.14 seems inconsistent with the CUDA compatibility documentation

Accelerated Computing CUDA CUDA Setup and Installation
, ,
1

Hi NVIDIA team,

I am trying to understand the expected behavior of CUDA forward compatibility on a host with:

  • GPU: NVIDIA A800-SXM4-80GB
  • Driver: 550.54.14
  • nvidia-smi reports CUDA 12.4

I am referring to the CUDA compatibility documentation here:

My confusion is that the practical behavior I observe does not seem easy to reconcile with the current documentation, especially around CUDA 13.x forward
compatibility and cuda-compat.

What I tested:

  1. Container with manually installed cuda-compat packages
  • cuda-compat-13.0 works
  • cuda-compat-13.1 works
  • cuda-compat-13.2 fails
  • cuda-compat-13.3 fails

This result is consistent across multiple PyTorch versions and both CUDA lines I tested:

  • cu130: PyTorch 2.11.0 / 2.12.1 / 2.13.0
  • cu132: PyTorch 2.12.0 / 2.12.1 / 2.13.0

The failure mode is consistently:

  • cudaGetDeviceCount()
  • Error 803: system has unsupported display driver / cuda driver combination
  1. NVIDIA official CUDA Ubuntu 22.04 images
    I also tested NVIDIA official CUDA images and manually installed PyTorch inside the container.

Observed results:

  • nvidia/cuda:13.0.0-base-ubuntu22.04 + torch 2.12.1+cu130: PASS
  • nvidia/cuda:13.1.0-base-ubuntu22.04 + torch 2.12.1+cu130: PASS
  • nvidia/cuda:13.2.0-base-ubuntu22.04: container startup is blocked by nvidia-container-cli
  • nvidia/cuda:13.3.0-base-ubuntu22.04 starts, but PyTorch CUDA initialization fails with Error 803
  • nvidia/cuda:13.3.0-base-ubuntu22.04 fails for both torch 2.12.1+cu130 and torch 2.12.1+cu132

Additional detail:

  • For 13.2.0-base, the failure happens before entering the container:

    • nvidia-container-cli: requirement error: unsatisfied condition: cuda>=13.2

  • For 13.3.0-base, the container can start, but CUDA initialization in PyTorch still fails with Error 803

So on this machine, the practical boundary looks like:

  • CUDA 13.0 / 13.1: usable
  • CUDA 13.2 / 13.3: not usable

My question:

Could NVIDIA please clarify what the expected forward compatibility status is for driver branch 550.54.14 with:

  • cuda-compat-13.0
  • cuda-compat-13.1
  • cuda-compat-13.2
  • cuda-compat-13.3

More specifically, I would like to understand:

  • Is 550.54.14 supposed to be a supported target for any CUDA 13.x forward compatibility configuration?
  • If yes, why do 13.2 and 13.3 fail in practice with Error 803?
  • If no, is the documentation missing an explicit statement about how the 550 driver branch should be interpreted?
  • Is this expected behavior, or does it indicate an issue in the documentation?

At the moment, the documentation leaves me unsure whether I am misunderstanding the support matrix, or whether the current page needs clarification for
the 550 driver branch.

If helpful, I can provide:

  • exact test commands
  • PyTorch versions
  • container image tags
  • full error logs

Thanks.

2

Can someone help me with this issue? I use the 550.54.14 driver and failed to use CUDA 13.2 and 13.3, but according to the docs( Forward Compatibility — CUDA Compatibility ), it says my driver is compatible with the newest CUDA version.

3

Looking at the document you cite:

"In addition to the CUDA driver and certain compiler components, there are other drivers in the system installation stack (for example, OpenCL) that remain on the old version. The forward-compatible upgrade path is for CUDA only." (emphasis mine).

"A well-written application should use following error codes to determine if a CUDA Forward Compatible Upgrade is supported.

.....

CUDA_ERROR_SYSTEM_DRIVER_MISMATCH = 803. This error indicates that there is a mismatch between the versions of the display driver and the CUDA driver."

If you feel this doesn’t cover the situation, you could file a bug.

4

Update:

In the CUDA forward compatibility document:

New Feature Branches are not supported targets for CUDA Forward Compatibility. This is why R550 doesn’t work for CUDA 13.x even if you install ‘cuda-compat-13-3’

about how to read the table:

For example, the column labeled “535+” (or R535) indicates driver versions that start with 535, such as 535.xx.xx. It does not mean any driver version numerically greater than 535 (like my 550.xx.xx).

So, to facilitate this scenario, any of the following should work:

  1. Update to the latest CUDA 13.3 released driver , R610 , use no compatibility
  2. Update to R580 drivers , use Enhanced compatibility
  3. Choose any of supported 535/570/580/590/595 branch driver and install cuda-compat-13-3 , use Forward compatibility
5

The key distinction in the CUDA compatibility doc is minor-version forward compatibility, not arbitrary forward compatibility. A CUDA 12.x application built with the newer minor toolkit can run on an older 12.x driver if the driver supports the PTX/SASS features used, but only within the same major version and only if the application does not use features introduced after the driver.

Driver 550.54.14 is a CUDA 12.4-era driver. If your application was built with CUDA 12.5+, it may call cubins or use driver APIs that 550 does not recognize, even though the major version is “12.” That manifests as cudaErrorNoKernelImageForDevice or runtime API failures that look like “inconsistent behavior.”

Practical rule:

- For a built application (SASS/cubin), the driver must support the exact SM architecture and the instruction set version in the cubin.

- For PTX JIT, the driver must contain a PTX compiler new enough for the PTX version in the code.

- “Forward compatible” means a 12.4 driver can run a 12.4-or-earlier app; it does not promise a 12.4 driver will run a 12.5 app.

If you need a single binary to span driver versions, ship PTX and test on the oldest driver you support. If you hit a specific API that behaves differently on 550 vs. 560, post the exact call and error code — that’s easier to explain concretely.