Hi, I’m trying to monitor the GPU utilization of dGPU on the AGX Pegasus platform. However, I’m having trouble finding a way to do this. I can monitor the iGPU using tegrastat utility, however according to Can't detect the dGPU utilization through tegrastat, it appears that tegrastats is not supported (and will not be supported) for the dGPU. It mentions to check nvprof or Nsight for dGPU utilization. I tried using nvprof and Nsight Compute and there are utilization stats for individual components of the GPU when running a process, but no overall GPU utilization. I heard (or read somewhere) that maybe Nsight Systems CLI would provide overall GPU utilization. However, I’m not able to find a version of Nsight Systems CLI that works with AGX platform. I tried using the Nsight Systems GUI, but it does not seem to provide this statistic. Any help would be appreciated. Thanks! Please provide the following info (check/uncheck the boxes after clicking “+ Create Topic”): Software Version DRIVE OS Linux 5.2.0 DRIVE OS Linux 5.2.0 and DriveWorks 3.5 NVIDIA DRIVE™ Software 10.0 (Linux) NVIDIA DRIVE™ Software 9.0 (Linux) other DRIVE OS version other Target Operating System Linux QNX other Hardware Platform NVIDIA DRIVE™ AGX Xavier DevKit (E3550) NVIDIA DRIVE™ AGX Pegasus DevKit (E3550) other SDK Manager Version 1.5.0.7774 other Host Machine Version native Ubuntu 18.04 other

[image] kenneth.chu: However, I’m not able to find a version of Nsight Systems CLI that works with PX2 platform. Hi @kenneth.chu , May I know this topic is for DRIVE PX 2 / AGX devkit? On your host system, you can try to find the nsight-system installed with DRIVE OS Linux 5.2.0 by SDK Manager. $ grep -ri nsight-systems ~/.nvsdkm/

It is for DRIVE AGX Pegasus devkit. I corrected the description in the original post. I can install Nsight System GUI from DRIVE OS Linux 5.2.0 and run it from the Linux host. However, I could not find the overall GPU utilization of the dGPU in the GUI. If it does has this capability, is there some documentation on how to get this info? I could not find a CLI version of Nsight Systems. Not sure if it’s true, but I’ve heard from somewhere else, that the Nsight System CLI does have overall GPU utilization.

It seems to be installed to /opt/nvidia/nsight-systems so you can try to find if cli and relevant document there. Thanks.

On the host, I can see the documentation in /opt/nvidia/nsight-systems/2020.3.3/documentation, but most of the files are empty (just a cover page). I found a reference to the docs at Nsight Systems Documentation though. Not sure this applies to this version of Nsight Systems. On the target, I can run /opt/nvidia/nsight_systems/nsys. But I still don’t see anywhere which will tell me the overall GPU utilization. Any ideas about this? I’ll keep looking in the mean time. Thanks.

Please go head and check Nsight Systems Documentation to see if anything you’re looking for. I think it won’t have much difference.

I couldn’t find anywhere that could tell me the overall GPU utilization on AGX Pegasus. Just wanted to confirm with you that there is no way to monitor the overall GPU utilization on Pegasus right now. Thanks.

Please check if API can measure or query values of performance counters helps.

I took a look at CUPTI and it looks like the metrics are the same (or similar) as the metrics from Nsight Compute. Is my understanding correct? I was able to run Nsight Compute on the processes running in our system. With Nsight Compute, I can capture the GPU Speed Of Light, Compute Workload Analysis and Memory Workload Analysis sections. However, the issue I have is that we have multiple processes using the GPU and I do not know how to combine these metrics from multiple processes into an overall GPU utilization. Ideally, I’d like the same information that is provided by nvidia-smi. Example: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 450.102.04 Driver Version: 450.102.04 CUDA Version: 11.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce RTX 208... Off | 00000000:01:00.0 Off | N/A | | 30% 30C P8 541W / 250W | 93MiB / 11019MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 2066 G /usr/lib/xorg/Xorg 91MiB | +-----------------------------------------------------------------------------+ Notice the GPU utilization, memory usage, etc. I understand that this tool is for PC and uses the NVML library to get this information. However, that the NVML library is not supported on ARM (Pegasus platform). Any ideas how I can get this same information on Pegasus? For example, how I can use the metrics from CUPTI or Nsight Compute to understand the overall GPU utilization?

Any more ideas about this? Please let me know either way whether it is possible using the current libraries and tools to measure the overall GPU utilization on AGX Pegasus. This is critical to us for our current project to understand how many loads we can run on the current platform. So far I’ve tried tegrastats, Nsight Compute and Nsight Systems and I haven’t been able find a way to measure the overall GPU utilization. For tegrastats, the utilization always shows 0%. For Nsight Compute, it appears that only a single process can be profiled at a time (according to the documentation, multiple processes are serialized) and for Nsight Systems, it appears that the GPU metrics on not enabled on this version. Thanks for your help and any ideas on how we can move forward.

Currently we don’t have a tegrastats-like tool to monitor dGPU usage. The nsys CLI tool supports an option ( --stats) that prints summary of CUDA kernel and memory operation statistics on the target. $nsys --stats=true profile This summary is closely related to GPU utilization. Alternatively the qdrep file obtained after profiling can also be used to obtain the stats using $nsys stats

Monitoring GPU utilization of dGPU on DRIVE AGX Pegasus

Autonomous Vehicles DRIVE AGX Xavier DRIVE AGX Xavier General

VickNV April 30, 2021, 8:56pm 14

Sorry that it should be:

$ nsys profile --stats=true

Please refer to Nsight Systems User Guide :: NVIDIA Nsight Systems Documentation in NVIDIA DRIVE DOCUMENTATION site.

Nsys command line on agx pegasus

Topic		Replies	Views
Using nsight to profile discrete gpu DRIVE AGX Xavier General drive-devtools	1	1163	September 22, 2021
How to get dGPU utilizations on Pegasus DRIVE AGX Xavier General drive-devtools , drive-admin	2	889	December 15, 2022
Can't detect the dGPU utilization through tegrastat General	31	6096	August 15, 2019
0% GPU usage from tegrastats in DRIVE OS 5.0.10.3L DPX2 General	8	1762	March 18, 2020
How to use dGPUs DRIVE AGX Xavier General driveos	9	1527	August 6, 2021
Nsys command line on agx pegasus Profiling DRIVE Targets drive-devtools	13	2256	November 16, 2021
Checking the GPU utilization with DRIVE AGX DRIVE AGX Xavier General	3	920	November 30, 2018
Feature request for average GPU utlization Profiling Linux Targets	7	984	April 24, 2019
Measure GPU utilization on Jetson Platforms Jetson Xavier NX profiling , nvidia-smi	8	266	August 28, 2026
Xavier GPU resources monitor Jetson AGX Xavier	7	908	November 12, 2020

Monitoring GPU utilization of dGPU on DRIVE AGX Pegasus

Related topics