No Permission to Performance Counters as Root

When running a simple example with ncu, I got this:
==ERROR== Error: ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM

Problem is, I am running this as root in a docker container, so I should have root access to the host machine.
All the solutions to this problem I have found so far involves granting access to non-root users, so they wouldn’t work because I am already root.

What else could be the problem?

Driver Version: 418.87.00
CUDA Version: 10.1
Nsight Compute Version: 2020.1.0
Card: Tesla V100-PCIE-32GB
OS: Ubuntu 16.04

1 Like

Any solutions? Similar issues here. I also tried following command line in host to grant permission to all users:

systemctl isolate multi-user
modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0
systemctl isolate graphical

Still not work.

OS: Ubuntu 16.04
Card: Tesla P100
CUDA Version: 10.2 (Host)
Driver version 440.95.01
Docker Image: Nvidia NGC nvcr.io/nvidia/tensorrt:20.02-py3 (cuda verison 10.2)