Hello, what effect does thread switching on the host have on tensorrt inference? I’ve found that on my server, using a multi-threaded inference model takes longer than a single-threaded inference time, which doesn’t include c’u’d’aCopy time, and G’P’U is less utilized;
Hi @2235636388 ,
Apologies for the delay, I am checking on this, and shall revert soon.
Thank you
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Multithread does not improve inference performance with tensorrt models | 2 | 1304 | May 11, 2021 | |
| tensorRT5 inference speed slown down in multithread application. | 2 | 1405 | December 3, 2019 | |
| Tensorrt Threads affect each other during multithreaded inference | 16 | 1920 | September 6, 2024 | |
| TensorRT inference result of one image don't keep the same in high qps | 1 | 679 | June 29, 2022 | |
| Optimal Trt inference using threads/processes for peoplenet model for | 1 | 1228 | July 30, 2021 | |
| Tensorrt multi gpu with multi threads | 1 | 1231 | February 18, 2022 | |
| Multiple threads running inference are causing a slowdown | 1 | 916 | August 1, 2023 | |
| Does TensorRT uses multiple cores for inference for an input? | 1 | 660 | April 13, 2020 | |
| Multithread inference | 3 | 1069 | June 22, 2021 | |
| Speeding up multi-threaded C++ program of TensorRT models | 7 | 1652 | February 20, 2025 |