Why nvidia-smi show different CUDA version for same driver version?

I am using some GPUs on Aliyun. I have 2 GPUs, both with same driver version, but nvidia-smi shows different CUDA versions. And if I use my CUDA toolkit 12 in the container with CUDA 11 , not supprisingly , it throws cudaErrorInsufficientDriver.
Can someone explain why the same driver version has different advised CUDA version?
My screenshots:


Now I know, I have more than one libcuda.so (driver runtime i guess) in my container, and nvidia-smi actually use libcuda.so to show the CUDA Version. and CUDA toolkit also use libcuda.so. older version of libcuda.so cannot support nvcc of CUDA12. It is a LD_LIBRARY_PATH problem.

1 Like