Jetson Xavier GPU/DLA load percentage via SYSFS

Hi!
I am searching for a way to get performance indicators in our C/C++ application. Since the NVML API is not available for embedded devices, how can we measure CUDA usage, GPU usage or DLA usage? Is tegrastats the only way to get this in C/C++? It’s pretty limited and does not output any info on DLA…

The Jetson dev guide mentions sysfs interfaces available, but GPU load percentage is only available through tegrastats or the Jetson Power GUI.
https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide

Thanks!

You can use an application like strace (“sudo apt-get install strace”) to monitor system calls, which in turn will tell you much of what you want.

tegrastats runs forever, and so you’ll have to use CTRL-c to stop after a few moments if logging normally. You could redirect stdout to /dev/null since the actual strace output (when not logged) is to stderr. To see an interesting (repeating) list of system calls:
sudo strace tegrastats 1>/dev/null

More interesting:
sudo strace tegrastats 2>&1 | egrep '^openat'

Note that the “openat()” system call return value is a file descriptor. If “= -1” it is an error (0 is stdin, 1 is stdout, 2 is stderr, and 3 is stdlog…the latter of which is just a buffered stderr, thus “2>&1” redirects both stderr and stdlog with stdlog having better performance).

To see 100 lines (which means as many 1 second pauses as needed for system calls plus actual output to reach 100 lines, about 2 lines of normal tegrastats, but varies depending on system):
sudo strace tegrastats 2>&1 | egrep '^openat' | head -n 100 | less -N

1 Like

Great thanks a lot, found it!
I didn’t know strace, it’s really useful!

$ sudo cat /sys/devices/gpu.0/load
29

There is a less used corollary, “ltrace”. “strace” watches system calls to the kernel. “ltrace” watches library calls. Between the two of them it covers every call the program might make outside of its own code.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.