How to Monitor Discrete GPU Workload/Usage on PX2?

Autonomous Vehicles DRIVE PX 2 General
1

I am running a DNN program on my DRIVE PX2 AutoChauffeur and would like to monitor the discrete GPU’s performance. Running

sudo tegrastats

on my PX2 continuously outputs a line like this:

RAM 2162/6660MB (lfb 825x4MB) cpu [14%@1987,14%@1987,14%@1987,14%@1987,14%@1987] EMC 0%@1600 GR3D 10%@1275 GR3D_PCI 74%@2607

GR3D represents the GPU engine, so is it possible that “GR3D_PCI” represents the discrete GPU engine? Or is there another way to determine the usage of the discrete GPU engine? Thanks very much.

2

Dear JerryHong,

Yes, “GR3D_PCI” means the discrete GPU engine.
This is only way to check dGPU usage so far.
There is a sample called deviceQuery if CUDA has been installed on your DPX2. Run this sample, it will list GPU usage info in detail:
cd /usr/local/cuda-8.0/samples/1_Utilities/deviceQuery
sudo make
sudo ./deviceQuery
Thanks.

3

Thank you very much! I truly appreciate the response.

4

Why I could only see the CPU info, not GPU info?
I am using Beta0.2.1.
sudo tegrastats
RAM 516/6933MB (lfb 1521x4MB) cpu [0%@1996,0%@2034,0%@2035,0%@1997,0%@1995,0%@1997] EMC 0%@1600

5

Dear ZHANGShu,

Could you please update DPX2 SDK version to Beta3.0?
Can not be shown dGPU info with tegrastats command on Beta0.2.1. Thanks.

6

Hi SteveNV

“GR3D_PCI 74%@2607”
GR3D_PCI is dGPU, so what does “2607” mean? Could you explain a little more about this? Thank you~

I run the deviceQuery, the output is like below, max freq of dGPU is 1290MHz. It seems like that
2607 in “GR3D_PCI 74%@2607” is not clock rate.

Device 0: “Graphics Device”
CUDA Driver Version / Runtime Version 9.0 / 9.0
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 3840 MBytes (4026466304 bytes)
( 9) Multiprocessors, (128) CUDA Cores/MP: 1152 CUDA Cores
GPU Max Clock rate: 1290 MHz (1.29 GHz)

7

Dear wangjian38,

GR3D should be actual GPU core frequcency, GR3D-PCI should be memory frequency. You can see GR3D and GR3D-PCI frequency are 1275, 2607 respectively. Thanks.

8

Hi SteveNV

Here is output of tegrastats in PX2
RAM 3270/6660MB (lfb 403x4MB) cpu [51%@1981,71%@2033,92%@2037,54%@1985,46%@1978,60%@1983] EMC 0%@1600 GR3D 0%@1275 GR3D_PCI 43%@2607

and deviceQuery:

Device 0: “Graphics Device”
CUDA Driver Version / Runtime Version 9.0 / 9.0
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 3840 MBytes (4026466304 bytes)
( 9) Multiprocessors, (128) CUDA Cores/MP: 1152 CUDA Cores
GPU Max Clock rate: 1290 MHz (1.29 GHz)

Device 1: “NVIDIA Tegra X2”
CUDA Driver Version / Runtime Version 9.0 / 9.0
CUDA Capability Major/Minor version number: 6.2
Total amount of global memory: 6668 MBytes (6991458304 bytes)
( 2) Multiprocessors, (128) CUDA Cores/MP: 256 CUDA Cores
GPU Max Clock rate: 1275 MHz (1.27 GHz)
Memory Clock rate: 1600 Mhz

Device 0 is dGPU @ 1290 MHz.
Device 1 is iGPU @ 1275 MHz.

It seems like that “tegrastats” show us only iGPU usage(GR3D 0%@1275) and dGPU mem bandwidth usage(GR3D_PCI 43%@2607).
So what should I do to get dGPU usage and iGPU mem bandwidth usage

9

Dear wangjian38,

We have found a problem with tegrastats and are checking.
By the way, I think you are not using the latest PDK version(5.0.5.0bL) for DPX2. Thanks.

10

Dear SteveNV,

Has the issue been resolved now?

11

Dear kenw,
If you are asking about checking dGPU utilization via tegrastats, this is not supported by tegrastats and we have updated the documentation.

12

Could you provide a link or a general indication of where in the documentation I can get the information?

13

Hi kemw,

Please find the tegrastats Utility information from:https://docs.nvidia.com/drive/active/5.0.10.3L/nvvib_docs/index.html#page/NVIDIA%2520DRIVE%2520Linux%2520SDK%2520Development%2520Guide%2FUtilities%2FAppendixTegraStats.html