tensorRT5 inference speed slown down in multithread application.

AI & Data Science Deep Learning (Training & Inference) TensorRT
#1

Hi,
I wrote a demo using tensorRT5.1.5.0 to test SSD inference speed. In single thread case, speed is fast. I change demo to multithread, and test speed again. The inference speed slow down compared with single thread. And GPU Occupancy rate also turn down. Any ideas? Thank you!

#2

May be related to this.

‘TensorRT supports multiple threads so long as each is used with a separate execution context.’

From: https://devtalk.nvidia.com/default/topic/1032481/jetson-tx2/how-to-run-trt-in-multithreading-/post/5252796/#5252796

#3

Hi,

Please refer below link for some suggestions for using TensorRT with multithread:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-best-practices/index.html#thread-safety

Thanks