Use multiple CUDA streams with multiple TensorRT models

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

Hi, all.
Is it faster to try multiple CUDA streams with multiple tensorrt models than to combine multiple tensorrt models into one?

3

Hi,

It depends on the use case.
If your model doesn’t occupy all GPU resources at once, multiple CUDA streams can benefit from parallel inference.

Thanks

1 Like
4

Your answer is very helpful to me. Thanks.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.