Multi-model parallel inferencing

FreedomLiX · March 31, 2023, 5:21am

how to parallel inferencing multi models ? demo code

AakankshaS · March 31, 2023, 5:37am

Hi,

The below links might be useful for you.

For multi-threading/streaming, will suggest you to use Deepstream or TRITON

For more details, we recommend you raise the query in Deepstream forum.

or

raise the query in Triton Inference Server Github instance issues section.

Thanks!

Topic		Replies	Views
How to Thread the prediction function provided in NVIDIA / TensorRT IntroNotebooks? TensorRT tensorrt , cuda , jetson-inference	1	362	July 1, 2022
Multithread does not improve inference performance with tensorrt models TensorRT tensorrt	2	1168	May 11, 2021
Documentation for multi-model serving with overcommit on Triton TensorRT python , inference-server-triton	1	671	April 24, 2023
Tensorrt Threads affect each other during multithreaded inference TensorRT tensorrt	16	1310	September 6, 2024
Multiple context and/or multithreading TensorRT	1	1213	March 24, 2022
TensorRT MultiThread with MultiGPU TensorRT	1	469	February 14, 2023
Run multiple model(engine) with tensorrt without deepstream TensorRT	1	1115	April 20, 2020
Not able to inference multiple input models using TRT TensorRT tensorrt , tensorflow , jetson-inference	1	436	August 12, 2021
Concurrent tensorRT engines TensorRT jetson	1	383	December 5, 2022
How to inference with tensorrt on multi gpus in python TensorRT	2	2098	April 9, 2021