Unable to get GPU profile on Linux with CAP_SYS_ADMIN enabled user

Hello, I’m trying to profile a simple CUDA Application using Visual Profiler, and I’m getting the following error when I try to examine the GPU profile:

nvprof log: /home/zarnick/nvvp_workspace/.metadata/.plugins/com.nvidia.viper/launch/1/nvprof_44492.log
==44492== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM
==44492== Warning: Some profiling data are not recorded.

However, when I look into the user pcap permissions, I can see the CAP_SYS_ADMIN:

$ capsh --print
Current: = cap_sys_admin+i
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read
Ambient set =
Securebits: 00/0x0/1’b0
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
secure-no-ambient-raise: no (unlocked)
uid=1000(zarnick) euid=1000(zarnick)
gid=1000(zarnick)
groups=4(adm),24(cdrom),27(sudo),29(audio),30(dip),46(plugdev),121(lpadmin),131(lxd),132(sambashare),139(libvirt),998(docker),1000(zarnick),1004(vboxusers)
Guessed mode: UNCERTAIN (0)

Following the link from the error, it clearly says that either the user must have the CAP_SYS_ADMIN enabled (which it is) or I need to update the module to allow everyone to profile on the GPU (not desired). Also, if I try with sudo I can profile normally.

BTW: This is a Kubuntu 20.04 machine.

Can someone help me out?

Best regards

If anyone is having this issue, I was able to solve it by adding the option on a file on /etc/modprobe.d, and rebuilding the initrd. Basically:

$ cat /etc/modprobe.d/nvidia-prof.conf 
options nvidia "NVreg_RestrictProfilingToAdminUsers=0"
$ sudo update-initramfs -c -k 5.4.0-70-generic
$ sudo update-grub
$ sudo /sbin/reboot

Still seems like overkill to me, but at least it’s working now!

1 Like