# Run multiple model(engine) with tensorrt without deepstream

**URL:** <https://forums.developer.nvidia.com/t/run-multiple-model-engine-with-tensorrt-without-deepstream/120156>\
**Category:** TensorRT\
**Created:** [April 17, 2020, 4:12pm UTC](https://forums.developer.nvidia.com/t/run-multiple-model-engine-with-tensorrt-without-deepstream/120156 "2020-04-17T16:12:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jalalmiry](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jalalmiry](https://forums.developer.nvidia.com/u/jalalmiry)\
**Post date:** [April 17, 2020, 4:12pm UTC](https://forums.developer.nvidia.com/t/run-multiple-model-engine-with-tensorrt-without-deepstream/120156/1 "2020-04-17T16:12:46Z")

</div>

## Description

Hi,  
how can configure multiple model (back to back pipeline) with tensorrt . (multi-thread). in this configuration how set context, allocate memory for multiple engine.  
in my configuration i take below error in memory :

boxes\_2, confs\_2, clss\_2 = trt\_yolov3.detect(img, conf\_th)  
File “/home/cv/Downloads/tensorrt\_demos-master/tensorrt\_demos-master/utils2/yolov3.py”, line 479, in detect  
stream=self.stream)  
File “/home/cv/Downloads/tensorrt\_demos-master/tensorrt\_demos-master/utils2/yolov3.py”, line 367, in do\_inference  
[cuda.memcpy\_htod\_async(inp.device, inp.host, stream) for inp in inputs]  
File “/home/cv/Downloads/tensorrt\_demos-master/tensorrt\_demos-master/utils2/yolov3.py”, line 367, in  
[cuda.memcpy\_htod\_async(inp.device, inp.host, stream) for inp in inputs]  
pycuda.\_driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument

A clear and concise description of the bug or issue.

## Environment

**TensorRT Version** :  
**GPU Type** :  
**Nvidia Driver Version** :  
**CUDA Version** :  
**CUDNN Version** :  
**Operating System + Version** :  
**Python Version (if applicable)**:  
**TensorFlow Version (if applicable)**:  
**PyTorch Version (if applicable)**:  
**Baremetal or Container (if container which image + tag)**:

## Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

## Steps To Reproduce

Please include:

- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered

---

<div class="post-metadata">

**Author:** ![SunilJB](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/suniljb/32/14043_2.png) [@SunilJB](https://forums.developer.nvidia.com/u/SunilJB)\
**Post date:** [April 20, 2020, 8:54am UTC](https://forums.developer.nvidia.com/t/run-multiple-model-engine-with-tensorrt-without-deepstream/120156/2 "2020-04-20T08:54:21Z")

</div>

Hi,

In order to run multiple model with TensorRT, i will recommend you to either use NVIDIA deepstream or NVIDIA Triton Inference Server.  
Please refer below link for more details:

> **[DeepStream SDK](https://developer.nvidia.com/deepstream-sdk)**
>
> Develop and deploy AI-powered intelligent video analytics apps and services faster anywhere.

> **[Documentation - Latest Release :: NVIDIA Deep Learning Triton Inference...](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/index.html)**
>
> This Triton Inference Server documentation focuses on the Triton inference server and its benefits. The inference server is included within the inference server container. This guide provides step-by-step instructions for pulling and running the...

If you want to perform multi threading using TensorRT, please refer below link for best practices:  
[https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-best-practices/index.html#thread-safety](https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-best-practices/index.html#thread-safety)

Thanks
