How to get GPU usage by an specific application(PID)

[diogojusten]

Dear All,
I’m trying to find a way to get how much GPU load a specific application is using. I can easily get the total GPU load by the whole system, but I would like to have from a specific process/PID only.
I’m running a Xavier AGX and can’t find the libnvidia-ml.so file (runtime file) for that board, also run CUDA 10.2.
I don’t want to get GPU memory, but GPU load, similar as exist for CPU load.
Does someone have any insight on how I can achieve it?
Bests,
Diogo

[greivin.brenes]

Hello @diogojusten

You may be looking for something like jtop you can install it by running:

sudo pip3 install -U jetson-stats

So you may want to run the tool with jtop in a terminal and go to the GPU section, there it will display a list of running processes along with their GPU utilization, among other information.

Hope this helps!
Regards!

Greivin Brenes
Embedded SW Engineer at RidgeRun

Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com

[diogojusten]

Hi @greivin.brenes,
thank you for your answer. jtop shows the global GPU usage.
E.g.: In a scenario where there are two application using GPU resources, I would like to know how much each application is using GPU. Similar top does for %CPU.
jtop shows the global how much the whole GPU is being used, plus in the GPU tab there is GPU memory by each process. It doesn’t provide the GPU load in %, similar for %CPU.

[AastaLLL]

Hi,

Have you tried our Nsigt Compute profiler?
You can get the used GPU resources in detail by profiling the target app:

Nsight Compute :: Nsight Compute Documentation

Thanks.

[diogojusten]

Hi @AastaLLL, thank you for your reply.
Yes, I could use Nsight tool to get GPU usage from a specific PID. Just giving a bit more of the context, I’m developing a runtime health monitor where it monitors HW resource usage by multiple other applications. As it is a runtime application, I can’t use the Nsight Tool for a continuous and automatic monitoring.

[AastaLLL]

Hi,

You can profile it with attach mode.
And level after collecting enough info.

Does this work for you?
Thanks.