TensorRT runtime batch processing in C++

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Could not able to process or set the batch size to TRT engine.

Environment

TensorRT Version: 8.0.1
GPU Type: GeForce
Nvidia Driver Version: 470.57.02
CUDA Version: 11.4
CUDNN Version: 8.2.*
Operating System + Version: Ubuntu 18.04 LST
Python Version (if applicable): NO
TensorFlow Version (if applicable): NO
PyTorch Version (if applicable): 1.8.1
Baremetal or Container (if container which image + tag): NO

Relevant Files

trt_v8_cpp_batch_processing.zip (20.6 KB)

Steps To Reproduce

  1. Run the model.py to generate a dummy model [Used ]
  2. Run the CMake build command to build the tensorrt_model
  3. ./tensorrt_model 0 → To serialize the TRT engine from ONNX model
  4. ./tensorrt_model 1 → To deserialize the TRT engine and check the CMD output

LOG

================= Engine Info =================
batch_size_engine : 2
num_bindings : 2
================= Input =================
        Name : input
        Shape : [1, 3, 20, 40, ]
================= Output =================
        Name : output
        Shape : [1, 8, 5, 10, ]
================= Context Info =================
Dimension Specified 1
        Input Shape : [1, 3, 20, 40, ]
        Output Shape : [1, 8, 5, 10, ]


================= After setting binding dimension to Context Info =================
3: [executionContext.cpp::setBindingDimensions::969] Error Code 3: Internal Error (Parameter check failed at: runtime/api/executionContext.cpp::setBindingDimensions::969, condition: profileMaxDims.d[i] >= dimensions.d[i]. Supplied binding dimension [2,3,20,40] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 1, minimum dimension in profile is 1, but supplied dimension is 2.
)
        Output Shape : [1, 3, 20, 40, ]

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2
  1. The first image in the batch is processing
  2. Second image in the batch is not

I am passing same image in batch 2

3

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

5

I have shared ONNX model in the zip file

6

I could able to achieve and run the TRT model with batch size 2.
Following are the steps that I followed

  1. As per your suggestion
    https://docs.nvidia.com/deeplearning/tensorrt/quick-start-guide/index.html#onnx-export

I have modified the ONNX_model for batch 2.
2. Build TRT model using the batch size in builder
3. Then context has batch size 2

After doing that I could see a difference in the result

  1. I have prepred a batch of 2 for same image
  2. Ran the model
  3. Image 1 in batch has propper output, however, image 2 some of the results varies.

I could not able to share the model because this model is internal.
If you could tell me something about this why two results varies it will very helpful.

7

Hi @anil2,

Looks like you have defined model input batch size as static (1), You may need to have dynamic input. Please refer following doc for more details.

Thank you.