tensorRT5 inference speed slown down in multithread application.

yanlei9875 · November 15, 2019, 10:04am

Hi,
I wrote a demo using tensorRT5.1.5.0 to test SSD inference speed. In single thread case, speed is fast. I change demo to multithread, and test speed again. The inference speed slow down compared with single thread. And GPU Occupancy rate also turn down. Any ideas? Thank you!

orcdnz · November 27, 2019, 11:24am

May be related to this.

‘TensorRT supports multiple threads so long as each is used with a separate execution context.’

From: https://devtalk.nvidia.com/default/topic/1032481/jetson-tx2/how-to-run-trt-in-multithreading-/post/5252796/#5252796

SunilJB · December 3, 2019, 6:18am

Hi,

Please refer below link for some suggestions for using TensorRT with multithread:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-best-practices/index.html#thread-safety

Thanks

Topic		Replies	Views
Does tensorRT5 or tensorRT6 support multithreading? TensorRT	2	849	November 27, 2019
Tensorrt Threads affect each other during multithreaded inference TensorRT tensorrt	16	1445	September 6, 2024
TensorRT inference is slower under multi-threading TensorRT tensorrt , cuda	1	413	January 31, 2024
Multithread does not improve inference performance with tensorrt models TensorRT tensorrt	2	1186	May 11, 2021
[TensorRT] engine happed a error in multithreaded TensorRT tensorrt , cuda	2	1568	January 19, 2023
Multithread inference Jetson Xavier NX tensorrt	4	839	August 29, 2021
Tensorrt multi gpu with multi threads TensorRT	1	1097	February 18, 2022
Speeding up multi-threaded C++ program of TensorRT models TensorRT tensorrt	7	1374	February 20, 2025
Multiple threads running inference are causing a slowdown TensorRT jetson , deepstream	1	765	August 1, 2023
How to use TensorRT by the multi-threading package of python Jetson AGX Xavier tensorrt	13	18803	October 18, 2021

tensorRT5 inference speed slown down in multithread application.

Related topics