# Not able to inference multiple input models using TRT

**URL:** <https://forums.developer.nvidia.com/t/not-able-to-inference-multiple-input-models-using-trt/186278>\
**Category:** TensorRT\
**Tags:** tensorflow, tensorrt, jetson-inference\
**Created:** [August 11, 2021, 10:27am UTC](https://forums.developer.nvidia.com/t/not-able-to-inference-multiple-input-models-using-trt/186278 "2021-08-11T10:27:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![reshu.singh](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@reshu.singh](https://forums.developer.nvidia.com/u/reshu.singh)\
**Post date:** [August 11, 2021, 10:27am UTC](https://forums.developer.nvidia.com/t/not-able-to-inference-multiple-input-models-using-trt/186278/1 "2021-08-11T10:27:42Z")

</div>

## Description

While doing the inference on multi-input models using the snippet below-

saved\_model\_dir\_trt =   
root = tf.saved\_model.load(saved\_model\_dir\_trt)  
infer(load\_test\_data)

where, (load\_test\_data) is the function which returns a list of preprocessed images to be fed into the  
model trained.

## Environment

**TensorRT Version** : 7.1.3  
**GPU Type** :  
**Nvidia Driver Version** : 460  
**CUDA Version** : 11.2  
**CUDNN Version** : 8.0.2  
**Operating System + Version** : Ubuntu18  
**Python Version (if applicable)**: 3.6.9  
**TensorFlow Version (if applicable)**: 2.4.0

---

<div class="post-metadata">

**Author:** ![NVES](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nves/32/14043_2.png) [@NVES](https://forums.developer.nvidia.com/u/NVES)\
**Post date:** [August 12, 2021, 5:35am UTC](https://forums.developer.nvidia.com/t/not-able-to-inference-multiple-input-models-using-trt/186278/2 "2021-08-12T05:35:29Z")

</div>

Hi,  
The below link might be useful for you  
[https://docs.nvidia.com/deeplearning/tensorrt/best-practices/index.html#thread-safety](https://docs.nvidia.com/deeplearning/tensorrt/best-practices/index.html#thread-safety)

> **[Programming Guide :: CUDA Toolkit Documentation](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#stream-priorities)**
>
> The programming guide to the CUDA model and interface.

[https://docs.nvidia.com/cuda/cuda-runtime-api/group\_\_CUDART\_\_STREAM.html](https://docs.nvidia.com/cuda/cuda-runtime-api/group __CUDART__ STREAM.html)  
For multi threading/streaming, will suggest you to use Deepstream or TRITON  
For more details, we recommend you to raise the query to the Deepstream or TRITON forum.

> **[DeepStream SDK](https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/deepstream-sdk/15)**
>
> Discussions about the DeepStream SDK

> **[Triton Inference Server - archived](https://forums.developer.nvidia.com/c/ai-data-science/deep-learning/triton-inference-server/97)**

Thanks!
