Work with batch in TensorRT


Hello everyone,

I’m new in using TensorRT Python API.
Could you help me to migrate simple angle prediction model from Keras framework to TensorRT via ONNX? Now the main trouble is batch processing. My model has one dynamic input (batch of images). I created TRT engine with trtexec:

./trtexec --explicitBatch --onnx=apm_one_input.onnx --minShapes=input:1x64x64x3 --optShapes=input:20x64x64x3 --maxShapes=input:100x64x64x3 --saveEngine=apm_one_input.plan

On the inference step model works well with 1 image, but not with batch. Output buffer is empty. This command:

give this result:

(1, 64, 64, 3)
So engine allocate memory for only 1 image. Could you help?


TensorRT Version:
GPU Type: GeForce GTX 1060 6 GB
Nvidia Driver Version: 440.33.01
CUDA Version: 10.2
CUDNN Version: 7.1
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable): 2.3.1

Hi @v.stadnichuk,

Could you please share issue reproducible inference script and model file for better assistance.

Thank you.

Hi @spolisetty ,
I sent you scripts and model via private message.

Thank you @v.stadnichuk, we will look into it.