Consistent Inference Time & GPU Frequency Stabilization After Initial Runs - Jetson AGX Orin / TensorRT FP32

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hi Nvidia Community,

I’m observing a curious behavior during TensorRT inference on a Jetson AGX Orin, and I’m hoping to gain some insights from the forum.

I’m running an FP32 TensorRT model on a dataset of 2075 images. The inference times consistently decrease after the first few runs, stabilizing around 101ms from the 3rd run onwards.

Here’s a summary of the observed average enqueue times:

image
image596×182 37.4 KB

The variance and standard deviation also decrease significantly after the 2nd run, indicating a much more stable performance once the inference settles. Specifically, the variance drops from 497.425 (Run 1) to under 2 (Runs 5 & 6).

Further investigation using tegrastats reveals a corresponding behavior in GPU frequency. During the initial runs, the GPU frequency fluctuates significantly. However, starting with the 3rd run, the GPU frequency consistently stabilizes around 611 MHz. (See attached tegrastats screenshot).

1st Run:

GR3D_FREQ (%) Value
93% [611, 0]
99% [407, 0]
0% [509, 0]
93% [611, 0]
99% [408, 0]
0% [509, 0]
54% [494, 0]
0% [488, 0]
0% [509, 0]
89% [471, 0]
0% [509, 0]
0% [509, 0]
2% [611, 0]
0% [509, 0]
0% [611, 0]
99% [407, 0]
99% [407, 0]
73% [505, 0]
38% [611, 0]

3rd Run onwards:

GR3D_FREQ (%) Value
47% [611, 0]
99% [605, 0]
33% [611, 0]
99% [611, 0]
78% [609, 0]
99% [610, 0]
28% [611, 0]
99% [611, 0]
35% [611, 0]
60% [607, 0]
99% [604, 0]
22% [611, 0]
76% [611, 0]
99% [611, 0]
99% [617, 0]
99% [607, 0]
78% [609, 0]
99% [611, 0]
99% [611, 0]
99% [611, 0]
99% [611, 0]
99% [611, 0]

My question is: Why is this behavior observed? Is this a common occurrence with TensorRT and the Jetson AGX Orin? As these execution happens right after one another, Is TensorRT heuristically deciding to increase the GPU frequency as there is a demand for GPU?

Below are the Current configuration for Jetson AGX Orin I am using:

image
image1143×556 199 KB

Any insights or suggestions for further investigation would be greatly appreciated.

Thank you!

4

Hi,

This is expected as the first run might need some extra initialization overhead.
Usually, we recommend having some warm-up steps to prevent the delay in the beginning.

https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-1030/developer-guide/index.html#trtexec-duration-iterations

Thanks.

5

Hi,

We are aware of initial warmup for few frames in every execution as it evident in each execution. However we want to understand that after some initial frames, the execution time gets decreases in consecutive runs. Here, each row represents single inference.

image
image221×765 27.9 KB

6

Hi,

Please set up the warm-up for at least 200 ms.
Thanks.