# \[TensorRT\] engine happed a error in multithreaded

**URL:** <https://forums.developer.nvidia.com/t/tensorrt-engine-happed-a-error-in-multithreaded/239708>\
**Category:** TensorRT\
**Tags:** tensorrt, cuda\
**Created:** [January 16, 2023, 6:08am UTC](https://forums.developer.nvidia.com/t/tensorrt-engine-happed-a-error-in-multithreaded/239708 "2023-01-16T06:08:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![devin.he](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@devin.he](https://forums.developer.nvidia.com/u/devin.he)\
**Post date:** [January 16, 2023, 6:08am UTC](https://forums.developer.nvidia.com/t/tensorrt-engine-happed-a-error-in-multithreaded/239708/1 "2023-01-16T06:08:45Z")

</div>

## Description

The problem shown as the attached file that when we loop our inference for several times, each image per loop, it always \*\*shows the strange result \*\*, when we use it in multithreaded.

Because we need to use it in TCP server (Python) and provide a real-time identify function.  
 ![image](https://global.discourse-cdn.com/nvidia/optimized/3X/f/8/f83f8c3e93f455cccd7560b49a50f2fdeb02eac5_2_690x63.png)

```auto

./receive230116022455/ac_receive.jpg
[01/16/2023-06:04:08] [TRT] [E] 1: [convolutionRunner.cpp::execute::391] Error Code 1: Cask (Cask convolution execution)
[01/16/2023-06:04:08] [TRT] [E] 1: [checkMacros.cpp::catchCudaError::272] Error Code 1: Cuda Runtime (invalid resource handle)
['./receive230116022455/ac_receive.jpg', array([0., 0., 0., 0.], dtype=float32), 'NG1']

```

It could get the right result, and the data file is right.  
And if reload the engine in single thread, the result may get the right result [like OK].

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/9/a/9a1384c555970c7768502f6dd960b354206aea4b.png)

```auto
./receive230116022455/ac_receive.jpg
['./receive230116022455/ac_receive.jpg', array([0.001507 , 0.01713044, 0.19878952, 0.782573], dtype=float32), 'OK']

```

## Environment

**TensorRT Version** : 8.2.5.1  
**GPU Type** : RTX5000  
**Nvidia Driver Version** : 515.65  
**CUDA Version** : 11.7  
**CUDNN Version** :  
**Operating System + Version** : ubuntu 22.02  
**Python Version (if applicable)**: python3.8

## Relevant Files

## Steps To Reproduce

The env was [download] by this way

```auto
docker run --runtime=nvidia -dit --rm --entrypoint "" nvcr.io/nvidia/tensorrt:22.05-py3 /bin/bash

```

I try to use init() and pop() to avoid **the trt in multithreaded** , but it was uselessful.

```auto
cuda.init()
device = cuda.Device(0)
ctx = device.make_context()

###########
#process()
###########
ctx.push()
my.infer()
ctx.pop()

```

How to avoid **the logger return the 0 value** in multithreaded? It puzzled me for a long time!

---

<div class="post-metadata">

**Author:** ![spolisetty](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/spolisetty/32/14047_2.png) [@spolisetty](https://forums.developer.nvidia.com/u/spolisetty)\
**Post date:** [January 17, 2023, 9:44am UTC](https://forums.developer.nvidia.com/t/tensorrt-engine-happed-a-error-in-multithreaded/239708/2 "2023-01-17T09:44:51Z")

</div>

Hi,

The below links might be useful for you.

> **[NVIDIA TensorRT Documentation — NVIDIA TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/latest/)**

> [@How to use TensorRT by the multi-threading package of python](https://forums.developer.nvidia.com/t/how-to-use-tensorrt-by-the-multi-threading-package-of-python/123085/8):
>
> Hi, The sample is ready. The workflow looks like this: Master: create TensorRT engine and buffer, store the created CUDA context. Thread1: restore master’s CUDA context, run inference, destroy its CUDA context. Master: destroy the stored CUDA context. The sample is created with TensorRT default MNIST model. 1. Prepare data $ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --saveEngine=mnist.trt $ cd /usr/src/tensorrt/data/mnist/ $ sudo pip3 install pillow $ pyt…

> **[cudaStreamCreateWithPriority() - CUDA C++ Programming Guide (Legacy) — CUDA C++...](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#stream-priorities)**
>
> Applications manage the concurrent operations described above through streams. A stream is a sequence of commands (possibly issued by different host threads) that execute in order. Different streams, on the other hand, may execute their commands out...

> **[CUDA Runtime API :: CUDA Toolkit Documentation](https://docs.nvidia.com/cuda/cuda-runtime-api/group __CUDART__ STREAM.html)**

For multi-threading/streaming, will suggest you to use Deepstream or TRITON

For more details, we recommend you raise the query in the Deepstream forum.

or

raise the query in the Triton Inference Server Github instance issues section.

> **[Triton Inference Server](https://github.com/triton-inference-server)**
>
> Triton Inference Server has 36 repositories available. Follow their code on GitHub.

Thanks!

---

<div class="post-metadata">

**Author:** ![devin.he](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@devin.he](https://forums.developer.nvidia.com/u/devin.he)\
**Post date:** [January 19, 2023, 6:58am UTC](https://forums.developer.nvidia.com/t/tensorrt-engine-happed-a-error-in-multithreaded/239708/3 "2023-01-19T06:58:00Z")

</div>

Tks！

I found the TensorRT is [Thread Safety](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-803/best-practices/index.html#thread-safety)！

```auto
The TensorRT builder may only be used by one thread at a time. If you need to run multiple builds simultaneously, you will need to create multiple builders.
The TensorRT runtime can be used by multiple threads simultaneously, so long as each object uses a different execution context.
Note: Plugins are shared at the engine level, not the execution context level, and thus plugins which may be used simultaneously by multiple threads need to manage their resources in a thread-safe manner. This is however not required for plugins based on IPluginV2Ext and derivative interfaces since we clone these plugins when ExecutionContext is created.
The TensorRT library pointer to the logger is a singleton within the library. If using multiple builder or runtime objects, use the same logger, and ensure that it is thread-safe.

```

So， I change the sever in **single thread** way and it works well.

And refer

> [@How to use TensorRT by the multi-threading package of python](https://forums.developer.nvidia.com/t/how-to-use-tensorrt-by-the-multi-threading-package-of-python/123085/8):
>
> Hi, The sample is ready. The workflow looks like this: Master: create TensorRT engine and buffer, store the created CUDA context. Thread1: restore master’s CUDA context, run inference, destroy its CUDA context. Master: destroy the stored CUDA context. The sample is created with TensorRT default MNIST model. 1. Prepare data $ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --saveEngine=mnist.trt $ cd /usr/src/tensorrt/data/mnist/ $ sudo pip3 install pillow $ pyt…

But when I try to use it in [multithreaded](https://forums.developer.nvidia.com/t/tensorrt-engine-happed-a-error-in-multithreaded/239708) way, it asks App to use the different GPU mem.

It didn’t work and send the same result.

```auto
./receive230116022455/ac_receive.jpg
[01/16/2023-06:04:08] [TRT] [E] 1: [convolutionRunner.cpp::execute::391] Error Code 1: Cask (Cask convolution execution)
[01/16/2023-06:04:08] [TRT] [E] 1: [checkMacros.cpp::catchCudaError::272] Error Code 1: Cuda Runtime (invalid resource handle)
['./receive230116022455/ac_receive.jpg', array([0., 0., 0., 0.], dtype=float32), 'NG1']

```
