What is the difference between `libcudart.so` present in different directories in linux?

Accelerated Computing CUDA CUDA Programming and Performance
1

I was a bit confused while trying to find the location of the libcudart.so file on my linux machine.

The path for libcuda.so (CUDA driver API library) is fixed and it occurs at only one place (if I am not mistaken):

/usr/lib/x86_64-linux-gnu$ ls -lh libcu*
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcublasLt.so -> libcublasLt.so.10
lrwxrwxrwx 1 root root   25 Sep  7 22:55 libcublasLt.so.10 -> libcublasLt.so.10.2.0.168
-rwxr-xr-x 1 root root  32M Sep  7 22:55 libcublasLt.so.10.2.0.168
-rw-r--r-- 1 root root  35M Sep  7 22:55 libcublasLt_static.a
lrwxrwxrwx 1 root root   15 Sep  7 22:55 libcublas.so -> libcublas.so.10
lrwxrwxrwx 1 root root   23 Sep  7 22:55 libcublas.so.10 -> libcublas.so.10.2.0.168
-rwxr-xr-x 1 root root  60M Sep  7 22:55 libcublas.so.10.2.0.168
-rw-r--r-- 1 root root  72M Sep  7 22:55 libcublas_static.a
lrwxrwxrwx 1 root root   29 Sep  9 21:13 libcudadebugger.so.1 -> libcudadebugger.so.535.104.05
-rwxr-xr-x 1 root root 9.8M Sep  9 21:13 libcudadebugger.so.535.104.05
lrwxrwxrwx 1 root root   12 Sep  9 21:13 libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root   21 Sep  9 21:13 libcuda.so.1 -> libcuda.so.535.104.05
-rwxr-xr-x 1 root root  28M Sep  9 21:13 libcuda.so.535.104.05
lrwxrwxrwx 1 root root   29 Sep  1 01:38 libcudnn.so -> /etc/alternatives/libcudnn_so
lrwxrwxrwx 1 root root   17 Sep  3 21:31 libcudnn.so.7 -> libcudnn.so.7.6.3
-rw-r--r-- 1 root root 410M Aug 15  2019 libcudnn.so.7.6.3
lrwxrwxrwx 1 root root   32 Sep  1 01:38 libcudnn_static.a -> /etc/alternatives/libcudnn_stlib
-rw-r--r-- 1 root root 387M Aug 15  2019 libcudnn_static_v7.a
-rw-r--r-- 1 root root  55K May 25 18:14 libcupscgi.so.1
lrwxrwxrwx 1 root root   23 May 15 20:05 libcupsfilters.so.1 -> libcupsfilters.so.1.0.0
-rw-r--r-- 1 root root 211K May 15 20:05 libcupsfilters.so.1.0.0
-rw-r--r-- 1 root root  34K May 25 18:14 libcupsimage.so.2
-rw-r--r-- 1 root root  26K May 25 18:14 libcupsmime.so.1
-rw-r--r-- 1 root root 115K May 25 18:14 libcupsppdc.so.1
-rw-r--r-- 1 root root 558K May 25 18:14 libcups.so.2
lrwxrwxrwx 1 root root   19 Mar 15  2023 libcurl-gnutls.so.3 -> libcurl-gnutls.so.4
lrwxrwxrwx 1 root root   23 Mar 15  2023 libcurl-gnutls.so.4 -> libcurl-gnutls.so.4.5.0
-rw-r--r-- 1 root root 515K Mar 15  2023 libcurl-gnutls.so.4.5.0
lrwxrwxrwx 1 root root   16 Mar 15  2023 libcurl.so.4 -> libcurl.so.4.5.0
-rw-r--r-- 1 root root 523K Mar 15  2023 libcurl.so.4.5.0

But when I tried to find the location of libcudart.so (CUDA Runtime API library), I found it at at least two places:

/usr/local/cuda/lib64$ ls -lh libcu*
-rw-r--r-- 1 root root 703K Sep  7 22:55 libcudadevrt.a
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcudart.so -> libcudart.so.10.1
lrwxrwxrwx 1 root root   21 Sep  7 22:55 libcudart.so.10.1 -> libcudart.so.10.1.168
-rwxr-xr-x 1 root root 493K Sep  7 22:55 libcudart.so.10.1.168
-rw-r--r-- 1 root root 868K Sep  7 22:55 libcudart_static.a
lrwxrwxrwx 1 root root   14 Sep  7 22:55 libcufft.so -> libcufft.so.10
lrwxrwxrwx 1 root root   20 Sep  7 22:55 libcufft.so.10 -> libcufft.so.10.1.168
-rwxr-xr-x 1 root root 132M Sep  7 22:55 libcufft.so.10.1.168
-rw-r--r-- 1 root root 137M Sep  7 22:55 libcufft_static.a
-rw-r--r-- 1 root root 143M Sep  7 22:55 libcufft_static_nocallback.a
lrwxrwxrwx 1 root root   15 Sep  7 22:55 libcufftw.so -> libcufftw.so.10
lrwxrwxrwx 1 root root   21 Sep  7 22:55 libcufftw.so.10 -> libcufftw.so.10.1.168
-rwxr-xr-x 1 root root 489K Sep  7 22:55 libcufftw.so.10.1.168
-rw-r--r-- 1 root root  33K Sep  7 22:55 libcufftw_static.a
lrwxrwxrwx 1 root root   18 Sep  7 22:55 libcuinj64.so -> libcuinj64.so.10.1
lrwxrwxrwx 1 root root   22 Sep  7 22:55 libcuinj64.so.10.1 -> libcuinj64.so.10.1.168
-rwxr-xr-x 1 root root 7.5M Sep  7 22:55 libcuinj64.so.10.1.168
-rw-r--r-- 1 root root  32K Sep  7 22:55 libculibos.a
lrwxrwxrwx 1 root root   15 Sep  7 22:55 libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root   21 Sep  7 22:55 libcurand.so.10 -> libcurand.so.10.1.168
-rwxr-xr-x 1 root root  58M Sep  7 22:55 libcurand.so.10.1.168
-rw-r--r-- 1 root root  58M Sep  7 22:55 libcurand_static.a
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcusolver.so -> libcusolver.so.10
lrwxrwxrwx 1 root root   23 Sep  7 22:55 libcusolver.so.10 -> libcusolver.so.10.1.168
-rwxr-xr-x 1 root root 175M Sep  7 22:55 libcusolver.so.10.1.168
-rw-r--r-- 1 root root  91M Sep  7 22:55 libcusolver_static.a
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcusparse.so -> libcusparse.so.10
lrwxrwxrwx 1 root root   23 Sep  7 22:55 libcusparse.so.10 -> libcusparse.so.10.1.168
-rwxr-xr-x 1 root root 117M Sep  7 22:55 libcusparse.so.10.1.168
-rw-r--r-- 1 root root 127M Sep  7 22:55 libcusparse_static.a

Again at :

/usr/local/cuda/targets/x86_64-linux/lib$ ls -lh libcu*
-rw-r--r-- 1 root root 703K Sep  7 22:55 libcudadevrt.a
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcudart.so -> libcudart.so.10.1
lrwxrwxrwx 1 root root   21 Sep  7 22:55 libcudart.so.10.1 -> libcudart.so.10.1.168
-rwxr-xr-x 1 root root 493K Sep  7 22:55 libcudart.so.10.1.168
-rw-r--r-- 1 root root 868K Sep  7 22:55 libcudart_static.a
lrwxrwxrwx 1 root root   14 Sep  7 22:55 libcufft.so -> libcufft.so.10
lrwxrwxrwx 1 root root   20 Sep  7 22:55 libcufft.so.10 -> libcufft.so.10.1.168
-rwxr-xr-x 1 root root 132M Sep  7 22:55 libcufft.so.10.1.168
-rw-r--r-- 1 root root 137M Sep  7 22:55 libcufft_static.a
-rw-r--r-- 1 root root 143M Sep  7 22:55 libcufft_static_nocallback.a
lrwxrwxrwx 1 root root   15 Sep  7 22:55 libcufftw.so -> libcufftw.so.10
lrwxrwxrwx 1 root root   21 Sep  7 22:55 libcufftw.so.10 -> libcufftw.so.10.1.168
-rwxr-xr-x 1 root root 489K Sep  7 22:55 libcufftw.so.10.1.168
-rw-r--r-- 1 root root  33K Sep  7 22:55 libcufftw_static.a
lrwxrwxrwx 1 root root   18 Sep  7 22:55 libcuinj64.so -> libcuinj64.so.10.1
lrwxrwxrwx 1 root root   22 Sep  7 22:55 libcuinj64.so.10.1 -> libcuinj64.so.10.1.168
-rwxr-xr-x 1 root root 7.5M Sep  7 22:55 libcuinj64.so.10.1.168
-rw-r--r-- 1 root root  32K Sep  7 22:55 libculibos.a
lrwxrwxrwx 1 root root   15 Sep  7 22:55 libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root   21 Sep  7 22:55 libcurand.so.10 -> libcurand.so.10.1.168
-rwxr-xr-x 1 root root  58M Sep  7 22:55 libcurand.so.10.1.168
-rw-r--r-- 1 root root  58M Sep  7 22:55 libcurand_static.a
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcusolver.so -> libcusolver.so.10
lrwxrwxrwx 1 root root   23 Sep  7 22:55 libcusolver.so.10 -> libcusolver.so.10.1.168
-rwxr-xr-x 1 root root 175M Sep  7 22:55 libcusolver.so.10.1.168
-rw-r--r-- 1 root root  91M Sep  7 22:55 libcusolver_static.a
lrwxrwxrwx 1 root root   17 Sep  7 22:55 libcusparse.so -> libcusparse.so.10
lrwxrwxrwx 1 root root   23 Sep  7 22:55 libcusparse.so.10 -> libcusparse.so.10.1.168
-rwxr-xr-x 1 root root 117M Sep  7 22:55 libcusparse.so.10.1.168
-rw-r--r-- 1 root root 127M Sep  7 22:55 libcusparse_static.a

I find the ld links the cuda programs against the libcudart.so present in /usr/local/cuda/targets/x86_64-linux/lib on my system. But I have seen few posts, where libcudart.so is also available at /usr/lib/x86_64-linux-gnu

2

Some of this will be dependent on the install method.

If you use an installer provided by NVIDIA here, then the CUDA libraries at /usr/local/cuda/lib64 are in many cases also available at /usr/local/cuda/targets/...

I don’t know the exact reasons for this, and in general I don’t think it matters which you link against (they should be identical). The CUDA installer did not always do this, and I think the reasons for this evolution are several, having to do with supporting CUDA builds in a variety of settings. One of them relates to the stubs library, which is present in the targets/… sub-section.

NVIDIA is not the only one who creates “installers” for CUDA. You can find installers in repositories maintained e.g. by canonical (for Ubuntu) and other places. In my experience these installers often do things differently, and may install in /usr/lib/… rather than /usr/local/… like NVIDIA CUDA installers do. I suspect that this lines up with the way other packages and installers work. For example, depending on the install method, you may find nvcc (or a symlink to it) in /usr/bin for example. NVIDIA installers of the type I indicated don’t do that, in my experience.

If we ignore what is in the stubs library, the libcuda.so is not installed by the CUDA installer, it is installed by the GPU driver installer. And yes, that typically only ends up in one place.

1 Like
3

@Robert_Crovella Thanks for your valuable insight.