No Permission to Performance Counters as Root

When running a simple example with ncu, I got this:
==ERROR== Error: ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM

Problem is, I am running this as root in a docker container, so I should have root access to the host machine.
All the solutions to this problem I have found so far involves granting access to non-root users, so they wouldn’t work because I am already root.

What else could be the problem?

Driver Version: 418.87.00
CUDA Version: 10.1
Nsight Compute Version: 2020.1.0
Card: Tesla V100-PCIE-32GB
OS: Ubuntu 16.04

1 Like

Any solutions? Similar issues here. I also tried following command line in host to grant permission to all users:

systemctl isolate multi-user
modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0
systemctl isolate graphical

Still not work.

OS: Ubuntu 16.04
Card: Tesla P100
CUDA Version: 10.2 (Host)
Driver version 440.95.01
Docker Image: Nvidia NGC nvcr.io/nvidia/tensorrt:20.02-py3 (cuda verison 10.2)

have you solve this problem?
I’m also confused about that.

For general information on running Nsight Compute in containers, you can check this blog post.

In your specific case, try adding

--cap-add=SYS_ADMIN 

to your docker command (as explained here for Nsight Systems).

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.