Profiling Python code using sudo

[amirfakhimbabaei]

When I use the following command to profile my python script:

nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s none -f true -x true --stats=true --cudabacktrace memory --cuda-memory-usage=true --gpu-metrics-device=0 --gpu-metrics-set=0 --gpu-metrics-frequency=1000 python test.py

I get this error:

GPU Metrics: The user running Nsight Systems does not have permission to access NVIDIA GPU Performance Counters on the target device. For more details, please visit https://developer.nvidia.com/ERR_NVGPUCTRPERM
- API function: NVPW_Device_PeriodicSampler_GetCounterAvailability(¶ms)
- Error code: 17
- Source function: static std::vector QuadDDaemon::EventSource::GpuMetricsBackend::Impl::CounterConfig::GetCounterAvailabilityImage(size_t)
- Source location: /build/agent/work/20a3cfcd1c25021d/QuadD/Target/quadd_d/quadd_d/jni/EventSource/GpuMetricsBackend.cpp:572

In the following link, it is mentioned that I should run it as root using sudo to solve this problem. But, when I do it like this:

sudo nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s none -f true -x true --stats=true --cudabacktrace memory --cuda-memory-usage=true --gpu-metrics-device=0 --gpu-metrics-set=0 --gpu-metrics-frequency=1000 python test.py

I get this error:

Program not found: python

What should I do to solve this problem?

[amirfakhimbabaei]

I tried to use --run-as option instead of sudo to run it as a user with root access, but I got the following error this time:

unrecognised option '--run-as'

Why is --run-as option not supported even though it is present in the latest nsys documentation?

[amirfakhimbabaei]

For those who have the same problem, I solved it by using python3 instead of python. Now, I have another problem! nsys does not recognize my conda environment, I don’t know how to do it yet. I will post the solution whenever I find it.

[hwilper]

@liuyis can you comment on the --run-as issue? And the first issue?

[liuyis]

Hi @amirfakhimbabaei, could you share the full command line for the --run-as error? What’s the Nsys version you were using?

Note that --run-as is used to launch the target application as a regular user when Nsys is being started as root, so you cannot use it to avoid using sudo for Nsys in this specific case.

Regarding to the Program not found: python error as well as the conda environment issue, could you please check if things work under sudo without Nsys? I.e. sudo python test.py in the case. You’ll need to make sure the environment is set up correctly for root user.

[amirfakhimbabaei]

@liuyis

Hi @amirfakhimbabaei, could you share the full command line for the --run-as error? What’s the Nsys version you were using?

Note that --run-as is used to launch the target application as a regular user when Nsys is being started as root, so you cannot use it to avoid using sudo for Nsys in this specific case.

As I stated above, I solved this problem by using python3 instead of python. Now I’m able to use sudo without any problem. Thanks for your clarification, I understand that using --run-as was not the correct solution even though I don’t understand why I get the unrecognised option '--run-as' error. Just for clarification, my nsys version is 2021.5.2.53-28d0e6e, and I use the following command line:

sudo nsys profile -t nvtx,cuda --run-as amir --stats=true --cudabacktrace memory --cuda-memory-usage=true --gpu-metrics-device=0 --gpu-metrics-set=0 --gpu-metrics-frequency=1000 python3 test.py

@liuyis

Regarding to the Program not found: python error as well as the conda environment issue, could you please check if things work under sudo without Nsys? I.e. sudo python test.py in the case. You’ll need to make sure the environment is set up correctly for root user.

By using python3, I do not get the Program not found: python error anymore. The only problem I have is with the conda environment. Using python3 test.py my program runs flawlessly, but when I use the following command:

sudo nsys profile -t nvtx,cuda --stats=true --cudabacktrace memory --cuda-memory-usage=true --gpu-metrics-device=0 --gpu-metrics-set=0 --gpu-metrics-frequency=1000 python3 test.py

I get ModuleNotFoundError: No module named 'torch' error (import torch is the first line of my code). This means that nsys is executing my code in the base environment where torch is to installed.

[liuyis]

I don’t understand why I get the unrecognised option '--run-as' error. Just for clarification, my nsys version is 2021.5.2.53-28d0e6e

--run-as switch is a new feature added since 2022.1. You were using 2021.5 so that was why you were seeing the error. You can download 2022.1 release from https://developer.nvidia.com/gameworksdownload#?dn=nsight-systems-2022-1-1.

By using python3 , I do not get the Program not found: python error anymore. The only problem I have is with the conda environment.

Please try the same thing for python3 - i.e. make sure sudo python3 test.py works before profiling it under Nsys. You may to set up necessary environments for root user.

[amirfakhimbabaei]

@liuyis

--run-as switch is a new feature added since 2022.1. You were using 2021.5 so that was why you were seeing the error. You can download 2022.1 release from Download Center | NVIDIA Developer.

That makes sense now.

@liuyis

Please try the same thing for python3 - i.e. make sure sudo python3 test.py works before profiling it under Nsys. You may to set up necessary environments for root user.

It was the same using sudo python3 test.py . So it’s not an nsys problem, it has to do with conda and sudo. Thank you for your help!