Dynamic batch axis of input shape is not -1 in TensorRT8?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

The dynamic batch dimension should be -1 in TensorRT.

Environment

TensorRT Version: 8.0.1.6

Relevant Files

I convert the retinaface (Pytorch_Retinaface/convert_to_onnx.py at master · biubug6/Pytorch_Retinaface · GitHub) to onnx with dynamic axes

Steps To Reproduce

  1. Add dynamic axes in onnx
    dynamic_ax = {'input': {0:'batch', 2: 'image_height', 3: 'image_width'}, 'output_1': {1: 'total_anchors_1'}, 'output_2': {1: 'total_anchors_2'}, 'output_3': {1: 'total_anchors_3'}}
    #  dynamic_ax = {'input': {0: 'batch'}}
    input_names = ["input"]
    output_names = ["output_1", "output_2", "output_3"]
    inputs = torch.randn(1, 3, args.long_side, args.long_side).to(device)

    torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False,
                                   input_names=input_names, output_names=output_names, dynamic_axes = dynamic_ax, opset_version=11)
  1. Convert onnx to tensorrt

Here is the head of the log

[TensorRT] VERBOSE: Adding network input: input with dtype: float32, dimensions: (-1, 3, -1, -1)
[TensorRT] VERBOSE: Registering tensor: input for ONNX tensor: input
...

log.txt (848.4 KB)

using tensorrt api,

        engine_bytes = open(weight, 'rb').read()

        #  with trt.Logger() as logger, trt.Runtime(logger) as runtime:
        with trt.Runtime(logger) as runtime:
            self._engine = runtime.deserialize_cuda_engine(engine_bytes)
        assert(self._engine)

        self.input_shapes = []
        for binding in self._engine:
            if self.engine.binding_is_input(binding):
                shape = tuple(self.engine.get_binding_shape(binding))
                print(shape)
                self.input_shapes.append(shape)

the shape is still (1,3,-1,-1), but the expected input shape should be (-1,3,-1-1)

Any idea?

Here is onnx FaceDetector.onnx (1.7 MB)

2

I found out when max batch size is set to > 1, the batch dimension would be -1.
but when the max batch size is 1, the batch dimension is not -1, is this a bug for tensorrt?

5

Hi @OnePieceOfDeepLearning,

It seems since optimization profile for max_batch =1 makes batch =1 for all opt options, hence it’s getting replaced with 1. But when you are trying max_batch > 1 it remains -1 to handle all possible batch dim dynamically.

Thanks