Nvprof warns "the path to CUPTI and CUDA Injection libraries are not set in LD_LIBRARY_PATH"

Hi, thanks for the help in advance.
I had been using nvprof for a while and it had been working without any problems. A few days ago I was trying to do MPS profiling and I got a segfault. After that, the non-MPS profiling mode somehow stopped working, giving this warning every time I run sudo /usr/local/cuda-11.0/bin/nvprof WORKLOAD:

======== Warning: The path to CUPTI and CUDA Injection libraries might not be set in LD_LIBRARY_PATH. By default, CUPTI library is installed in /usr/local/<cuda-toolkit>/extras/CUPTI/lib64 or /usr/local/<cuda-toolkit>/targets/<arch>/lib and CUDA Injection library is installed in /usr/local/<cuda-toolkit>/targets/<arch>/lib
======== Warning: No profile data collected.

However, I have made sure that both paths are present in LD_LIBRARY_PATH before running nvprof, yet the warning is still given:

$ echo $PATH
/usr/local/cuda-11.0/bin:/home/ruipan/anaconda3/bin:/home/ruipan/anaconda3/condabin:/usr/local/cuda-11.0/extras/CUPTI/lib64:/usr/local/cuda-11.0/bin:/home/ruipan/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ echo $LD_LIBRARY_PATH
/usr/local/cuda-11.0/extras/CUPTI/lib64:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda-11.0/lib64

$ /usr/local/cuda-11.0/bin/nvprof printenv | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/usr/local/cuda-11.0/extras/CUPTI/lib64:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda-11.0/lib64:/usr/local/cuda-11.0/lib64:/usr/local/cuda-11.0/lib64
======== Warning: The path to CUPTI and CUDA Injection libraries might not be set in LD_LIBRARY_PATH. By default, CUPTI library is installed in /usr/local/<cuda-toolkit>/extras/CUPTI/lib64 or /usr/local/<cuda-toolkit>/targets/<arch>/lib and CUDA Injection library is installed in /usr/local/<cuda-toolkit>/targets/<arch>/lib
======== Warning: No profile data collected.

Also I want to add that my workload does not have any problem running without nvprof. Can someone please point me to what setup I am not configuring correctly? I would really appreciate the help.

=============================
Environment
CUDA: 11.0
CUDA driver version: 450.51.05
nvprof version: 11.0.194 (21)
OS: Linux 4.15.0-135-generic x86_64

my guess would be that this is due to the use of sudo. sudo does not necessarily pick up the PATH and LD_LIBRARY_PATH variables that you expect.

1 Like