Hi,
I am doing a research experiment and therefore got access to a departments private compute cluster, because I need to grid search optimal parameters.
The experiment requires JAX that supports cuda-12. Each node runs Driver Version 520.61.05 and CUDA Version 11.8.
Therefore I created a apptainer image that installs the cuda-compatability package version for CUDA-12.6
I installed it exactly the way it showed for the ubuntu documentation and added /usr/local/cuda-12.6/compat to LD_LIBRARY_PATH.
nvidia-smi shows the wanted cuda version of 12.6 using the image, however when trying to run the experiment I get: “failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_SYSTEM_DRIVER_MISMATCH: system has unsupported display driver / cuda driver combination”
“kernel version 520.61.5 does not match DSO version 560.35.3 – cannot find working devices in this configuration”
Does anyone know what the issue could be and how I could fix it?
Thanks in advance!