GPU/NVENC/NVDEC/VIC load units

[bhagen55]

Hello,
I would like to log the usage of the GPU, encoder, decoder, and VIC.

I believe tegrastats is reading from these nodes on the Orin NX:

/sys/devices/gpu.0/load
/sys/devices/platform/host1x/154c0000.nvenc/load
/sys/devices/platform/host1x/15480000.nvdec/load
/sys/devices/platform/host1x/15340000.vic/load

The VIC is the only one that lists a max load, and it is 900 on my device.
Otherwise, I don’t know what the unit of the numbers is. Is it frequency? Percentage?

I have seen the decoder load go between 0 and ~2000.

Can you help me determine how to calculate the overall load on these devices?

I would also like to record the load on the EMC, which tegrastats reports, but it is not immediately clear to me from the strace where it reads it from.

[DaneLLL]

Hi,
We would suggest run sudo tegrastats and check the prints. The tegrastats binary is prebuilt and not open source.

[bhagen55]

Hi,
Yes I am running tegrastats and trying to figure out where it is getting it’s information from. tegrastats’ output is not easily machine parse-able, and I need to record load metrics via my own task to ensure we don’t run out of GPU processing space at a critical point in a mission.

I know an nvidia representative hasn’t endorsed this directly, but many posts on the forum recommend using strace to figure out what tegrastats is reading from so we can access the information directly.

GPU usage info (nvidia-smi is not there)

The interesting thing about tegrastats is that it simply reads files in “/sys” or “proc”. As an interesting experiment, if you don’t already have “strace” installed, do so with “sudo apt-get install strace”. Then run tegrastats via strace, but redirect the “standard output” to “/dev/null”: sudo strace tegrastats 1>/dev/null You’ll see just the strace output, including the calls to files in “/proc” and “/sys”. This repeats, so just hit CTRL-c to stop after one reading. You can easily read those…

How to programmatically query CPU load?

I don’t know the details, but programs like ps, xosview, and top (and my favorite, htop) tend to aggregate what they use from “/proc”. Note that most of those link to “libprocps.so”, although they might use their own algorithm to look at individual files in “/proc”. An interesting tool is “strace”. For example, to see system calls from ps (mixed with actual ps output): sudo strace ps 2>&1 | less Within less search for “proc” via: /proc (then the “n” key for the next match, or “p” key for pr…

Source for tegrastats and/or info about querying overall GPU utilization?

GPU load reporting is at /sys/devices/platform/host1x/gpu.0/load For those looking for other points of data you can get more info by running sudo sh -c “strace ./tegrastats 2>&1 | grep open” open("/sys/devices/system/cpu/cpu1/online", O_RDONLY) = 3 open("/sys/devices/system/cpu/cpu2/online", O_RDONLY) = 3 open("/sys/devices/system/cpu/cpu0/online", O_RDONLY) = -1 ENOENT (No such file or directory) open("/sys/kernel/debug/clock/cpu/rate", O_RDONLY) = 3 open("/sys/devices/system/cpu/cpu1/online"…

[dusty_nv]

@bhagen55 jetson-stats / jtop has an API or you can browse the source to see the /sys and /proc files that it queries the device information from:

GitHub - rbonghi/jetson_stats: 📊 Simple package for monitoring and control...

[bhagen55]

Ok, I will get what information I can from the source code of that project.

We would really like a deeper API to fetch performance information from the GPU. We will be running multiple tasks that will likely be competing for GPU resources, and it seems like there is no way to record stats that would let us understand if/when we run out of cycles/bandwidth/etc.

Having the NVML API available would be fantastic, and probably fulfill all our needs. But it appears that is not supported on Tegra platforms.