#  Can TensorRT do inference in a child thread ?

**URL:** <https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925>\
**Category:** TensorRT\
**Created:** [June 5, 2019, 9:10am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925 "2019-06-05T09:10:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![longzhu\_71](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@longzhu\_71](https://forums.developer.nvidia.com/u/longzhu_71)\
**Post date:** [June 5, 2019, 9:10am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/1 "2019-06-05T09:10:38Z")

</div>

Hello,  
I have used tensorrt to inference my net,

as:  
with open(“my.engine”, “rb”) as f, trt.Runtime(TRT\_LOGGER) as runtime:  
engine = runtime.deserialize\_cuda\_engine(f.read())  
# Build a TensorRT engine.  
#with build\_engine\_onnx(onnx\_model\_file) as engine:  
# Allocate buffers and create a CUDA stream.  
h\_input, d\_input, h\_output, d\_output, stream = self.allocate\_buffers(engine)  
# Contexts are used to perform inference.  
with engine.create\_execution\_context() as context:  
# do inference

when I do this in the main thread,it works well.  
but when I start a Child thread to do this, it return a error as:

Exception in thread Thread-1:  
Traceback (most recent call last):  
File “/usr/lib/python3.6/threading.py”, line 916, in \_bootstrap\_inner  
self.run()  
File “/usr/lib/python3.6/threading.py”, line 864, in run  
self.\_target(\*self.\_args, \*\*self.\_kwargs)  
File “resnet50\_trt.py”, line 149, in prediction  
h\_input, d\_input, h\_output, d\_output, stream = self.allocate\_buffers(engine)  
File “resnet50\_trt.py”, line 56, in allocate\_buffers  
h\_input = cuda.pagelocked\_empty(trt.volume((INPUT\_SIZE,3,224,224)), dtype=trt.nptype(ModelData.DTYPE))  
pycuda.\_driver.LogicError: explicit\_context\_dependent failed: invalid device context - no currently active context?

same of the code:  
import pycuda.driver as cuda

```
# Allocate host and device buffers, and create a stream.
def allocate_buffers(self,engine):
    # Determine dimensions and create page-locked memory buffers (i.e. won't be swapped to disk) to hold host inputs/outputs.
    h_input = cuda.pagelocked_empty(trt.volume((INPUT_SIZE,3,224,224)), dtype=trt.nptype(ModelData.DTYPE))
    h_output = cuda.pagelocked_empty(trt.volume((INPUT_SIZE,8)), dtype=trt.nptype(ModelData.DTYPE))
    # Allocate device memory for inputs and outputs.
    d_input = cuda.mem_alloc(h_input.nbytes)
    d_output = cuda.mem_alloc(h_output.nbytes)
    # Create a stream in which to copy inputs/outputs and run inference.
    stream = cuda.Stream()
    return h_input, d_input, h_output, d_output, stream

```

the error seem happend at Allocate host and device buffers.  
So,Is the tensorRT can’t use in child thread?

---

<div class="post-metadata">

**Author:** ![longzhu\_71](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@longzhu\_71](https://forums.developer.nvidia.com/u/longzhu_71)\
**Post date:** [June 6, 2019, 5:24am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/2 "2019-06-06T05:24:51Z")

</div>

My code is base on the python samples in introductory\_parser\_samples onnx\_resnet50.py,I just change the code start a child thread ,and do inference in the child thread.

def main(name,times):  
# Set the data path to the directory that contains the trained models and test images for inference.  
data\_path, data\_files = common.find\_sample\_data(description=“Runs a ResNet50 network with a TensorRT inference engine.”, subfolder=os.path.join(“samples”, “resnet50”), find\_files=[“binoculars.jpeg”, “reflex\_camera.jpeg”, “tabby\_tiger\_cat.jpg”, ModelData.MODEL\_PATH, “class\_labels.txt”])  
# Get test images, models and labels.  
test\_images = data\_files[0:3]  
onnx\_model\_file, labels\_file = data\_files[3:]  
labels = open(labels\_file, ‘r’).read().split(‘\n’)

```
# Build a TensorRT engine.
with build_engine_onnx(onnx_model_file) as engine:
    # Inference is the same regardless of which parser is used to build the engine, since the model architecture is the same.
    # Allocate buffers and create a CUDA stream.
    h_input, d_input, h_output, d_output, stream = allocate_buffers(engine)
    # Contexts are used to perform inference.
    with engine.create_execution_context() as context:
        # Load a normalized test case into the host input page-locked buffer.
        test_image = random.choice(test_images)
        test_case = load_normalized_test_case(test_image, h_input)
        # Run the engine. The output will be a 1D tensor of length 1000, where each value represents the
        # probability that the image corresponds to that label
        do_inference(context, h_input, d_input, h_output, d_output, stream)
        # We use the highest probability as our prediction. Its index corresponds to the predicted label.
        pred =labels[np.argmax(h_output)]
        if "_".join(pred.split()) in os.path.splitext(os.path.basename(test_case))[0]:
            print("Correctly recognized " + test_case + " as " + pred)
        else:
            print("Incorrectly recognized " + test_case + " as " + pred)

```

if **name** == ‘ **main** ’:

```
action_classify = threading.Thread(target=main,args=('get-result',2),daemon=True)
action_classify.start()
#main()
print("main do same things")
while 1:
    pass

```

I got the same error .

I need do other things in main thread,so I need start a child thread to use tensorrt do inference.

Who can help me?

---

<div class="post-metadata">

**Author:** ![longzhu\_71](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@longzhu\_71](https://forums.developer.nvidia.com/u/longzhu_71)\
**Post date:** [June 10, 2019, 1:12am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/3 "2019-06-10T01:12:04Z")

</div>

Has anyone else encountered the same problem as me?

---

<div class="post-metadata">

**Author:** ![longzhu\_71](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@longzhu\_71](https://forums.developer.nvidia.com/u/longzhu_71)\
**Post date:** [June 13, 2019, 1:23am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/4 "2019-06-13T01:23:11Z")

</div>

If the tensorrt only can inference in the main thread, when I need do some UI things, how can I do?

---

<div class="post-metadata">

**Author:** ![longzhu\_71](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@longzhu\_71](https://forums.developer.nvidia.com/u/longzhu_71)\
**Post date:** [June 23, 2019, 9:40am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/5 "2019-06-23T09:40:01Z")

</div>

I used the cuda.Context.attach() ,then the inference can run .

---

<div class="post-metadata">

**Author:** ![luisyin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@luisyin](https://forums.developer.nvidia.com/u/luisyin)\
**Post date:** [August 23, 2019, 10:17am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/6 "2019-08-23T10:17:26Z")

</div>

Hi, longzhu\_71.Do you solve this problem? I am in same situation.

---

<div class="post-metadata">

**Author:** ![13509889178](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@13509889178](https://forums.developer.nvidia.com/u/13509889178)\
**Post date:** [August 11, 2020, 10:27am UTC](https://forums.developer.nvidia.com/t/can-tensorrt-do-inference-in-a-child-thread/75925/8 "2020-08-11T10:27:42Z")

</div>

> [@How to use TensorRT by the multi-threading package of python](https://forums.developer.nvidia.com/t/how-to-use-tensorrt-by-the-multi-threading-package-of-python/123085):
>
> Hi all, Purpose: So far I need to put the TensorRT in the second threading. I have read [this document](https://docs.nvidia.com/deeplearning/sdk/tensorrt-best-practices/index.html#streaming) but I still have no idea how to exactly do TensorRT part on python. I already have a sample which can successfully run on TRT. Now I just want to run a really simple multi-threading code with TensorRT. (I have done to generate the TensorRT engine, so I will load an engine and do TensorRT inference by multi-threading.) Here is my code below. (Without the Tensorrt code) import threading im…
