Human pose detection model (MoveNet) TensorRT conversion on NVIDIA Jetson

Robotics & Edge Computing Jetson Systems Jetson Xavier NX
, ,
7

Hello AastaLL,

Using trtexec and running the profiling tests, we get the follwing:

FP16

trtexec --loadEngine=trt_model/mv_lightning_fp16.engine

[06/06/2022-16:53:26] [I] === Performance summary ===
[06/06/2022-16:53:26] [I] Throughput: 497.66 qps
[06/06/2022-16:53:26] [I] Latency: min = 1.8024 ms, max = 15.8726 ms, mean = 1.96301 ms, median = 1.90088 ms, percentile(99%) = 2.04663 ms
[06/06/2022-16:53:26] [I] End-to-End Host Latency: min = 1.80756 ms, max = 15.8884 ms, mean = 1.97097 ms, median = 1.90775 ms, percentile(99%) = 2.17896 ms
[06/06/2022-16:53:26] [I] Enqueue Time: min = 0.938721 ms, max = 16.2527 ms, mean = 1.16295 ms, median = 1.02881 ms, percentile(99%) = 2.83414 ms
[06/06/2022-16:53:26] [I] H2D Latency: min = 0.0211182 ms, max = 0.0859375 ms, mean = 0.0240434 ms, median = 0.0229492 ms, percentile(99%) = 0.0281372 ms
[06/06/2022-16:53:26] [I] GPU Compute Time: min = 1.77563 ms, max = 15.7936 ms, mean = 1.93673 ms, median = 1.87494 ms, percentile(99%) = 2.02173 ms
[06/06/2022-16:53:26] [I] D2H Latency: min = 0.00146484 ms, max = 0.332764 ms, mean = 0.0022402 ms, median = 0.00180054 ms, percentile(99%) = 0.0022583 ms
[06/06/2022-16:53:26] [I] Total Host Walltime: 3.00406 s
[06/06/2022-16:53:26] [I] Total GPU Compute Time: 2.89541 s

INT8

trtexec --loadEngine=trt_model/mv_lightning_int8.engine

[06/06/2022-16:57:09] [I] === Performance summary ===
[06/06/2022-16:57:09] [I] Throughput: 592.577 qps
[06/06/2022-16:57:09] [I] Latency: min = 1.43872 ms, max = 30.7449 ms, mean = 1.65716 ms, median = 1.56552 ms, percentile(99%) = 2.74823 ms
[06/06/2022-16:57:09] [I] End-to-End Host Latency: min = 1.45007 ms, max = 30.7653 ms, mean = 1.66662 ms, median = 1.57373 ms, percentile(99%) = 2.83728 ms
[06/06/2022-16:57:09] [I] Enqueue Time: min = 0.93457 ms, max = 31.7563 ms, mean = 1.19099 ms, median = 1.02924 ms, percentile(99%) = 3.43677 ms
[06/06/2022-16:57:09] [I] H2D Latency: min = 0.0237732 ms, max = 1.07031 ms, mean = 0.0273041 ms, median = 0.0261841 ms, percentile(99%) = 0.0429688 ms
[06/06/2022-16:57:09] [I] GPU Compute Time: min = 1.41165 ms, max = 30.6662 ms, mean = 1.62093 ms, median = 1.53691 ms, percentile(99%) = 2.61005 ms
[06/06/2022-16:57:09] [I] D2H Latency: min = 0.00195312 ms, max = 10.9327 ms, mean = 0.00892758 ms, median = 0.00244141 ms, percentile(99%) = 0.0265503 ms
[06/06/2022-16:57:09] [I] Total Host Walltime: 3.00383 s
[06/06/2022-16:57:09] [I] Total GPU Compute Time: 2.88525 s

In both cases the mean is below 2ms, which means that the inference rate should be ~100Hz. However when I run the models using Python and the tensorrt libraries, I get about 11~13 FPS using either the FP16 or INT8 models. The performance is still not great.

Python Script

movenet_singlepose_trt.py is main python file which loads the engine and input file (video) from which the frames are derived and the inference is performed.

  • In line 18, you should be able to set the path to the TRT model.
  • The function getPoseFromVideo(video_path) is used to load a video using OpenCV and run the inference on the video frames.
  • Place the file FPS.py inside utils/

movenet_singlepose_trt.py (7.0 KB)
FPS.py (957 Bytes)

Models:
model_float32.onnx (9.0 MB)
mv_lightning_fp16.engine (7.7 MB)
mv_lightning_int8.engine (5.1 MB)

Is there anything that I’m not doing correctly in this script?

Thanks a lot!