I tried to profile ResNets(50;101;152) under MPS on A100.
Nsys works, however, since I need the utilization info, I tried NCU.
Command I used is :
ncu -o profile --target-processes all --set full python ../resnet/resnet/resnet50.py
Error returned is:
==ERROR==ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0.
Warning returned is:
==WARNING== No kernels were profiled.
Is this caused by the lack of permission to access NV GPU Perf Counters?
I saw this solution before, and I find it already created on the server. However, it doesn’t work for me.
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 NCU version: 2021.2.2.0 (build 30282580) (public-release)
Please note that profiling with Nsight Compute on MPS is not supported, as listed in the Known Issues.
With respect to the permissions error, it is not clear if this is caused by MPS or not. I suggest to try without MPS and if it persists, confirm the name/file extension and contents of the .conf file. E.g., the quotes need to be standard quote characters, sometimes they are copied differently from the website.
In general, I also recommend to use the latest version of Nsight Compute, since the tool is backwards-compatible. It won’t have an effect on the MPS support, though.
I had this issue, using an older geForce 1080. Swapped in a more recent newer GPU, RTX3070, no issues. It performed the profile fine:
==PROF== Profiling “addKernel” - 0: 0%…50%…100% - 34 passes
Note that GeForce 1080 has a Pascal GPU (GP102). Nsight Compute does not support Pascal GPU Architecture. Refer GPU support section of the Nsight Compute Release Notes.