TensorRT : Does TensorRT interpret the first dimension of layers shapes as batch size?

[juliefraysse]

Hi,

I have a theoretical question concerning the management of batch sizes in tensorRT.

In imported neural network models, should the first dimension of shapes for each layer of the neural network be dedicated to batch size? and, does TensorRT interpret it?

I ask myself this question because I would like to speed up a model initially created on tensorflow. I exported it in onnx format with tf2onnx.
In tensorflow, for each layer, the first dimension of the shape corresponds to the batch size.
This extra dimension is set to None to tell tensorflow that the batch size is not fixed.
Because the first dimension of the shapes is undefined, TensorRT considers the model with “dynamic shapes”.
However, I do not have the use of it: I would like to create my network with implicit batch size to be able to specify a bacth size at the time of the execute / inqueue function. (You can’t have a implicit batch if you have dynamic shapes.)

Environment:

** TensorRT Version **: 7.2
** CUDA Version **: 11.2
** TensorFlow Version (if applicable) **: 2.2

NB: I also noticed that in the onnx format, only the input and the output have dim_param at “unknow” or -1 whereas in tensorflow, all the layers have a first dimension of shape set to None.

[spolisetty]

Hi @juliefraysse,

ONNX model feed as explicit batch in TensorRT. So dynamic shape is a must for this.
Following link may help you.

TensorRT 7 ONNX models with variable batch size

Hi, I was previously using TRT5. My application was using different batch size (1,2,3,4 or 5) depending on a configuration parameter. I am now migrating to TRT 7. I read in multiple forums that the batch size must be explicit when parsing ONNX models in TRT7. How should I solve this using TRT7? Should I have different engines for each batch size?

Thank you.