Unable to profile with NCU -- WARNING: No Kernels were profiled

Development Tools Nsight Compute
, , ,
#1

I tried to profile ResNets(50;101;152) under MPS on A100.
Nsys works, however, since I need the utilization info, I tried NCU.

Command I used is :
ncu -o profile --target-processes all --set full python ../resnet/resnet/resnet50.py

Error returned is:
==ERROR==ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0.

Warning returned is:
==WARNING== No kernels were profiled.

Is this caused by the lack of permission to access NV GPU Perf Counters?
I saw this solution before, and I find it already created on the server. However, it doesn’t work for me.

NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 NCU version: 2021.2.2.0 (build 30282580) (public-release)

#2

Please note that profiling with Nsight Compute on MPS is not supported, as listed in the Known Issues.

With respect to the permissions error, it is not clear if this is caused by MPS or not. I suggest to try without MPS and if it persists, confirm the name/file extension and contents of the .conf file. E.g., the quotes need to be standard quote characters, sometimes they are copied differently from the website.

In general, I also recommend to use the latest version of Nsight Compute, since the tool is backwards-compatible. It won’t have an effect on the MPS support, though.