Hardware resources monitoring

[axnet]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson AGX Orin
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 5.1

I have developed a Deepstream pipeline and I’m trying to allocate tasks to different hardware parts of the Orin (DLA,VIC) . i wanted to know if there’s any way to monitor the usage load/power of every hardware part of the device other than powerGUI . PowerGUI doesn’t give clear results to monitor and it doesn’t provide data for DLA power consumption/VIC.
And is there a way to view the whether a specific task is running on CUDA core or Tensor Core?

[miguel.taylor]

Hi @axnet

You can use tegrastats:

sudo tegrastats

You can also use jtop for a nicer UI. The tool is installed with pip (It needs superuser privileges to install):

sudo pip3 install -U jetson-stats

You will need to reboot the board after installation and then run:

jtop

It has several tabs with different information:

1. ALL: Shows a summary of all the information that you can get with tegrastats but with a nicer UI
2. GPU: Shows GPU usage and shared memory occupation
3. CPU: Shows CPU usage over time for each core
4. MEM: System memory occupation over time and other memory information
5. ENG: The state of all the additional HW units (DLA, VIC, …)
6. CTRL: Change the profile. operation mode, and run jetson clocks
7. INFO: Hardware information and library versions

[axnet]

Thank you @miguel.taylor !! This really helps.
Do you have any idea if i can capture the output results for a specific time the way it is in powerGUI. I want to record the avg usage of hardware parts when running a sample video in DS pipeline

[miguel.taylor]

You can export the tegrastats output to a logfile using the --logfile option. Sometimes, we employ a Python thread to capture the tegrastats output.

def run_tegrastats():
    """
    Executes the tegrastats application as a subprocess.
    Updates the stats_current_output variable with the last line of tegrastats output.

    Raises
    ------
    AttrAccessError
        Error is triggered if the output can't be read
    """
    logging.debug("Running tegrastats...")
    global stats_current_output
    global stats_active
    args = ["tegrastats"]
    popen = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE)
    while stats_active:
        try:
            stats_current_output = popen.stdout.readline().decode('ascii')
        except Exception as exception:
            log_exception("Attribute access error",
                          "Invalid Tegrastats Output Error",
                          AttrAccessError)

A different process then parses the output using regex. You could implement something like this to check the stats as part of your Python app.