Libnvrtc.so is not using rpath to open libnvrtc-builtins.so

Accelerated Computing CUDA CUDA Programming and Performance
1

I am using TF-TRT with CUDA 11.0. I found that libnvrtc.so does not use rpath when opening “libnvrtc-builtins.so”.

Below is the strace of my program:

You see that libnvrtc only searches for libnvrtc-builtins in some system folders. But my program uses bazel to build and has a separate runfiles folder, which contains both “libnvrtc.so” and “libnvrtc-builtins.so”

[pid 8556] open(“tls/x86_64/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“tls/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“x86_64/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) = 23
[pid 8556] fstat(23, {st_mode=S_IFREG|0644, st_size=137517, …}) = 0
[pid 8556] mmap(NULL, 137517, PROT_READ, MAP_PRIVATE, 23, 0) = 0x7fdc9f1da000
[pid 8556] close(23) = 0
[pid 8556] access(“/etc/ld.so.nohwcap”, F_OK) = -1 ENOENT (No such file or directory)
[pid 8556] open(“/usr/local/cuda-11.0/targets/x86_64-linux/lib/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“/lib/x86_64-linux-gnu/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“/usr/lib/x86_64-linux-gnu/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“/lib/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] open(“/usr/lib/libnvrtc-builtins.so”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8556] munmap(0x7fdc9f1da000, 137517) = 0
[pid 8556] write(2, “nvrtc: error: failed to open lib”…, 113nvrtc: error: failed to open libnvrtc-builtins.so.
Make sure that libnvrtc-builtins.so is installed correctly.
) = 113

2

Hi hzhao, i hit the same issue and seems like yes, libnvrtc.so is not relying on rpath to find the libnvrtc-builtins.so. It’s using ld.so cached path, which is update when you install or install cuda. But if you really need it to use rpath, you can use patchelf to change the rpath of libnvrtc.so (patchelf --set-rpath “$ORIGIN” libnvrtc.so). This is how i get around the error.