When using TensorRT to optimize the inference running speed of the two estimators of nvidia Foundationpose

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

When using TensorRT to optimize the inference running speed of the two estimators of foundationpose, namely the score and pose refine estimators, it was found that the effect was not significant. The inference running time was reduced from over 2 seconds to 1.4 seconds. I don’t know what the reason is.

IMG_20251029_135631
IMG_20251029_1356311920×2560 275 KB

IMG_20251029_135659
IMG_20251029_1356591920×2560 272 KB

3

When the program starts running, the pose estimation time for the entire estimator is approximately 20 seconds, which is indeed quite slow.

4

Hi,

The first run is expected to be slow since some initialization is required.
Have you tried the command below to maximize the device performance first?

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

Thanks.

1 Like
5

These two instructions are useful. The time has been reduced from 20 seconds to around 6 seconds. Are there any other methods that we can discuss?

6

Hi,

Yes, for Jetson Nano, you can quantize the model to FP16 to get a better performance.
Usually, this can be done via trtexec with an extra configuration:

$ /usr/src/tensorrt/bin/trtexec --onnx=[file] --fp16

Thanks.

7

Yes，I did it.

8

Hi,

Please check if the GPU utilization is full in your use case first:

$ sudo tegrastats

Ideally, the loading of GR3D should be 99% when the device is running in performance mode.
Thanks.

9

However, the result I obtained was that the PT model took approximately 0.6 seconds (for an input size of 252*6*160*160), while the TensorRT result was around 0.2 seconds, and the reduction in time was not significant. However, there was a roughly 10-fold difference for Yolo and Detr.

10

Yes，it is OK. It should be sudo tegrastats and the GR3D is 99%

11

However, the result I obtained was that the PT model took approximately 0.6 seconds (for an input size of 252*6*160*160), while the TensorRT result was around 0.2 seconds, and the reduction in time was not significant. However, there was a roughly 10-fold difference for Yolo and Detr.

How about this？

12

Hi,

Sorry for the typo.

Do you use the same model for PT and TensorRT?
Suppose you can optimize both model with TensorRT.

Thanks.

13

Yes, of course.

14

However, sometimes I found it was not 99%. How to improve it when I am running the code?

15

Hi,

You can try to profile the pipeline with Nsight System and locate the bottleneck.
Usually, this is caused by some CPU-based pre-processing and post-processing.

Thanks.