Ncu-ui executable application python issue

Hi.
I have installed the nsight-comptue and tried to run stable diffusion application by giving command line arguments.
As I use pip virtual environment to run the application, I set the executable python path in venv_sd/bin/python. But it seems there is an error while handling python.

Also, if I do not specify the working directory, it automatically links to the /usr/bin/python, even I have already declared the application path.

Thank you !

Hi, @je.yang

Thanks for using Nsight Compute ! Please try with “Profile” activity instead.

Thanks for the answer.
Unfortunately, I still have same error while I have tried with activity - profile.

Thanks! Can you please paste the output printed in “Process Log” or you can use command line tool “ncu XXX” directly to check what error reports.

I think this may related with your virtual env, can you tell how to set up this env and we can have a try internally ?

:~/Desktop/2024_DiffusionModel$ ncu XXX
==ERROR== 'XXX' does not exist or is not an executable. Please make sure to specify the absolute path to 'XXX' if the executable is not in the local directory.

I generated the virtual env with the command “python -m venv venv_sd” to install the requirements for running stablediffusion2 (GitHub - Stability-AI/stablediffusion: High-Resolution Image Synthesis with Latent Diffusion Models).

For XXX, please replace with your own application and arguments. Before that, please also make sure that your application works separately without ncu.

The first part shows that application works separately with the argument while not calling ncu. With ncu, the application works well but it shows =ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM error.

  1 # This file was generated by nvidia-driver-535
  2 # Set value to 0 to disable modesetting
  3 options nvidia-drm modeset=1
  4 options NVreg_RestrictProfilingToAdminUsers=1                                                   

So I have tried to enable access permanently with the line 4 in nvidia-graphics-driviers-kms.conf at /etc/modeprobe.d, while it still brings same error and no kernel profiling.

Hi, @je.yang

Thanks for trying.
Please note it should be “options nvidia NVreg_RestrictProfilingToAdminUsers=0” in the conf file under /etc/modprobe.d. And you should reboot the machine to make it take effect.

For quick check, please just use “sudo ncu” instead of “ncu”

Following the guidance provided, I have successfully confirmed that profiling is taking place. I appreciate your assistance in this regard.

However, I am still having an issue where the application profiled through NVIDIA Profiler does not complete within the expected time and continues infinitely.
Additionally, I am curious about how to interpret the results of this profiling, I intended to use ncu-ui for this, but the UI shows the same error.

It is good to see the profiler works in your env now.
To reduce the profile time, you can use several options.

For example, -c to specify the kernel count. --kernel-name to specify the kernel you want to profile. Details please refer “ncu --help” .

NCU GUI also provide these filters.

Is there any way that I can check memory chart (transfer size, hit rates…) and roofline chart for overall applications using the command? As I cannot run the application through the UI yet, I need to use the command while it just shows the launch statistics per kernel not the entire applications.

Hi, @je.yang

You can generate the report from command line. And then open the report in NCU-UI.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.