ERR_NVGPUCTRPERM Despite following instructions

Developer Tools Nsight Compute
1

Hi,

I’m trying to use ncu to profile my GPU, but I keep running into the ERR_NVGPUCTRPERM permission denied error.

I have followed all steps at https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters and they do not work.

I’m using a RTX 3080 with CUDA 11.2, Driver 460.73.01 on Ubuntu 20.04.

Thanks!

2

The steps on the webpage are known to work, you might need to verify that they were applied in the correct order. Can you check the following:

Does profiling work for you when run as sudo/root?

Have you created a file with this exact content

options nvidia "NVreg_RestrictProfilingToAdminUsers=0"

within /etc/modprobe.d, e.g. as below

/etc/modprobe.d/nvidia.conf

and rebooted afterwards?

If so, have you confirmed that the file is still present after reboot?

2 Likes
3

I am not the original poster but the above instructions do work. On a side note, one doesn’t need to reboot. Instead:

  1. Stop the nvidia persistence deamon if that’s running:
    sudo systemctl stop nvidia-persistenced
  2. Unload the nvidia module (after unloading all the dependent modules that it would complain about). It’s usually:
    sudo rmmod nvidia_drm nvidia_uvm nvidia_modeset nvidia
  3. Insert the nvidia module: sudo modprobe nvidia

This will ensure the module is inserted with the option described upthread and ncu works without any issues post that.