Hey, I have the following error when starting nvcr.io/nvidia/cuda-dl-base:25.03-cuda12.8-devel-ubuntu24.04 container, but nvidia-smi on the host shows CUDA version is 12.8
What causes that ?
How CUDA/display drivers work between host and container and what compatibility constraints exist?
hgx-isr1-pre-01 ~ ➜ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
➜ enroot start cuda-dl-base-25.03-cuda12.8-devel-ubuntu24.04
==========
== CUDA ==
==========
NVIDIA Release (build )
CUDA Version 12.8.1.012
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available. [[ System has unsupported display driver / cuda driver combination (error 803) ]]