Running Multiple TensorRT Engine Error On Jetson Nano

[user162971]

Hi, I want to run two TensorRT Engine sequentially on Jetson Nano at the same process, then I reference this sample, update this code and try to test it. But the Bus error (core dumped) message will occur when python script finish. (sametimes the error message is “Segmentation fault (core dumped)”)

warm_up->(640, 640, 3), time->79.99ms
warm_up->(640, 640, 3), time->80.28ms
input->['samples/zidane.jpg'], time->81.50ms, saving into output/
input->['samples/zidane.jpg'], time->81.37ms, saving into output/
input->['samples/bus.jpg'], time->82.25ms, saving into output/
input->['samples/bus.jpg'], time->82.09ms, saving into output/
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1854, GPU 3881 (MiB)
Bus error (core dumped)

Update python code below

    yolov5_wrapper = YoLov5TRT(engine_file_path)
    yolov5_w2 = YoLov5TRT(engine_file_path)
    try:
        print('batch size is', yolov5_wrapper.batch_size)
        
        image_dir = "samples/"
        image_path_batches = get_img_path_batches(yolov5_wrapper.batch_size, image_dir)

        for i in range(10):
            # create a new thread to do warm_up
            thread1 = warmUpThread(yolov5_wrapper)
            thread1.start()
            thread1.join()
            thread2 = warmUpThread(yolov5_w2)
            thread2.start()
            thread2.join()
        for batch in image_path_batches:
            # create a new thread to do inference
            thread1 = inferThread(yolov5_wrapper, batch)
            thread1.start()
            thread1.join()
            thread2 = inferThread(yolov5_w2, batch)
            thread2.start()
            thread2.join()
    finally:
        # destroy the instance
        yolov5_wrapper.destroy() 
        yolov5_w2.destroy()

How can I avoid this error ?

thanks !

[kayccc]

Sorry for the late response, have you managed to get issue resolved or still need the support? Thanks

[AastaLLL]

Hi,

Jetson doesn’t support concurrent GPU buffer access.
Please find below for a sample to use TensorRT with multi-threads:

How to use TensorRT by the multi-threading package of python

Hi, The sample is ready. The workflow looks like this: Master: create TensorRT engine and buffer, store the created CUDA context. Thread1: restore master’s CUDA context, run inference, destroy its CUDA context. Master: destroy the stored CUDA context. The sample is created with TensorRT default MNIST model. 1. Prepare data $ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --saveEngine=mnist.trt $ cd /usr/src/tensorrt/data/mnist/ $ sudo pip3 install pillow $ pyt…

Thanks.

[user162971]

Hi

Thank your answer.
But I need to host two model sametimes and run sequentially in my Jetson Nano system.
This referenece only use one model.

I had try to use torchscript to inferenece model successfully.
Next, I want to save memory and speed up inference time, so I hope TensorRT runtime can be more efficent.

If you have any note please let me know

Thank you

[AastaLLL]

Hi,

Do the two models apply to the same input image?
If not, it’s recommended to create two processes for each.

For Nano, you can try to infer the model with fp16 mode for performance.
What kind of model you are using? Detector, classifier, or others?

Thanks.