libcupti.so not found after CUDA 10.1 Update 1

Accelerated Computing HPC Compilers Legacy PGI Compilers (archived)
1

After upgrading to CUDA 10.1 Update 1, I’m receiving the following error.

libcupti.so not found

I also received this error with CUDA 10.1 but was able to fix it with export below:

export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Even though libcupti.so is in the same location after the update, the error persists.

I’m using PGI_ACC_TIME=1 to profile OpenACC code.

The code can be found at https://github.com/mnicely/computeWorks_examples/tree/master/computeWorks_mm.

Add you can see if the following out the device code is not begin profiled.

Accelerator Kernel Timing data

  _Z7openACCifPKfS0_fPfRKi  NVIDIA  devicenum=0
    time(us): 43,743
    163: compute region reached 1000 times
        169: kernel launched 1000 times
            grid: [1x128]  block: [128]
            elapsed time(us): total=71,907 max=164 min=20 avg=71
    163: data region reached 2000 times
        163: data copyin transfers: 2000
             device time(us): total=26,742 max=19 min=13 avg=13
        178: data copyout transfers: 1000
             device time(us): total=17,001 max=22 min=13 avg=17

Matt

2

This issue is cause due to increased permission when using NVIDIA driver 418.67+

You can solve one of two ways.

  1. Run command with sudo (make sure you include the LD_LIBRARY_PATH)
sudo LD_LIBRARY_PATH=/usr/local/cuda/extra/CUPTI/lib64:$LD_LIBRARY_PATH ./computeWorks_mm
  1. Following Administration instructions @ https://developer.nvidia.com/nvidia-development-tools-solutions-ERR_NVGPUCTRPERM-permission-issue-performance-counters#SolnAdminTag 
sudo systemctl isolate multi-user # Stop the window manager




sudo su # Switch to root




modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia # Unload dependent modules




cd /etc/modprobe.d/




touch nvidia.conf # Create file named nvidia.conf




echo -e "options nvidia "NVreg_RestrictProfilingToAdminUsers=0"" > nvidia.conf




reboot