I wrote a demo using tensorRT188.8.131.52 to test SSD inference speed. In single thread case, speed is fast. I change demo to multithread, and test speed again. The inference speed slow down compared with single thread. And GPU Occupancy rate also turn down. Any ideas? Thank you!
May be related to this.
‘TensorRT supports multiple threads so long as each is used with a separate execution context.’
Please refer below link for some suggestions for using TensorRT with multithread: