TensorRT multithread hang

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I use the following procedure to run TensorRT in multiple pthread.

  1. Creating mContext[idx], where idx is from 0 to 3.
    |int tensorNet::createExecutionContext(int idx) {
    |nvinfer1::IExecutionContext* context = mEngine- >createExecutionContext();
    mContext[idx] = context;
    return 0;
    }

  2. Executing TensorRT context concurrently in pthreads
    int net::Detect(int idx) {
    void* inferenceBuffers[3] = {mInputCUDA, mOutputs[0].CUDA, mOutputs[1].CUDA};
    printf(“(%d) I E\n”, idx);
    if (!mContext[idx]->execute(1, inferenceBuffers)) {
    printf(LOG_TRT “net::Detect() – failed to execute TensorRT context\n”);
    return -1;
    }
    printf(“(%d) I X\n”, idx);
    }

  3. phread with idx=1 execute TensorRT but hang. The log messages is as follows.
    (1) E
    (1) X
    (0) E
    (0) X
    (1) E
    (1) X
    (0) E
    (0) X
    (1) E (pthread with idx=1 hang in mContext[idx]->execute(1, inferenceBuffers))
    (0) E
    (0) X

Do you have any suggestion to solve TensorRT multithread hang issue?

Environment

**TensorRT Version: 7.1.3
**GPU Type: 384-core NVIDIA Volta™ GPU with 48 Tensor Cores (Jetson NX, Jetpack 4.4)
CUDA Version: 10.2
CUDNN Version: 8.0
Operating System + Version: 4.9.140-tegra

2

Hi @andy.ueng,
Request you to check the below links for TensorRT with multithread here:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-best-practices/index.html#thread-safety

Thanks!

3

Thanks for your reply! Does Nvidia provide sample code for TensorRT multhreading?