Trtexec built engine with a fixed single batch output while dynamic input is allowed

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I have used trtexec to build engine from an onnx model with dynamic input size (-1,3,-1,-1), however the output is binded with batch size 1, while dynamic input is allowed. I was able to feed input with batch > 1, but always got output of batch=1. I am wondering that was due to the custom plugin I used. Please kindly help me figure it out.

The command line I used was:
./trtexec --onnx=model.onnx --saveEngine=model.engine --verbose --staticPlugins=/home/schen/Work/mmdeploy/lib/libmmdeploy_tensorrt_ops.so --minShapes=input:1x3x512x512 --optShapes=input:2x3x1024x1024 --maxShapes=input:3x3x1024x1024

Poloygraph analysis:

./polygraphy inspect model model.engine --plugins /home/schen/Work/mmdeploy/lib/libmmdeploy_tensorrt_ops.so

[W] ‘colored’ module is not installed, will not use colors when logging. To enable colors, please install the ‘colored’ module: python3 -m pip install colored
[I] Loading plugin library: /home/schen/Work/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[I] Loading bytes from model.engine
[W] TensorRT was linked against cuDNN 8.9.0 but loaded cuDNN 8.5.0
[I] ==== TensorRT Engine ====
Name: Unnamed Network 0 | Explicit Batch Engine

---- 1 Engine Input(s) ----
{input [dtype=float32, shape=(-1, 3, -1, -1)]}

---- 2 Engine Output(s) ----
{dets [dtype=float32, shape=(1, 2000, 6)],
 labels [dtype=int32, shape=(1, 2000)]}

---- Memory ----
Device Memory: 143233536 bytes

---- 1 Profile(s) (3 Tensor(s) Each) ----
- Profile: 0
    Tensor: input           (Input), Index: 0 | Shapes: min=(1, 3, 512, 512), opt=(2, 3, 1024, 1024), max=(3, 3, 1024, 1024)
    Tensor: dets           (Output), Index: 1 | Shape: (1, 2000, 6)
    Tensor: labels         (Output), Index: 2 | Shape: (1, 2000)

---- 451 Layer(s) ----

Environment

TensorRT Version: 8.6.1.6
GPU Type: NVIDIA_GeForce_RTX_3050_Ti_Laptop_GPU
Nvidia Driver Version: 550.54.15
CUDA Version: 12.4
CUDNN Version:
Operating System + Version: Ubuntu 22.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
1 Like
2

Hi @schen1 ,
Can you please help us with the model so that we can try it on our end?

3

Hi AakankshaS,

Thanks for you reply.

Please see the onnx model at:

You also need the plugin file at: