[TensorRT] batch vs shapes

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

I have exported my Torch model with the following command:

    torch.onnx.export(
        model, dummy_input, outfile, verbose=verbose,
        input_names=['input_batch'], output_names=output_names,
        # keep_initializers_as_inputs=True,
        opset_version=11,
        do_constant_folding=True,
        dynamic_axes={
            'input_batch': {0: 'dynamic'},
            'cif': {0: 'dynamic'},
            'caf': {0: 'dynamic'},
        },
    )

The shape of input is, (1x3x369x641, CHW)

The saved model is openpifpaf_resnet50_641_369.onnx

Now, I want to convert it to TensorRT engine file, with the batch size of 32.

I won’t change the width and height of input, but am interested in batch inference.

Which one is the correct command to use?

trtexec --onnx=openpifpaf_resnet50_641_369.onnx \
    --verbose \
    --explicitBatch \
    --minShapes=input_batch:1x3x369x641 \
    --maxShapes=input_batch:32x3x369x641 \
    --optShapes=input_batch:32x3x369x641 \
    --saveEngine=openpifpaf-resnet50-dynamic_b32.onnx \
    --fp16 \
    --workspace=16000
    
trtexec --onnx=openpifpaf_resnet50_641_369.onnx \
    --verbose \
    --explicitBatch \
    --batch=32 \
    --saveEngine=openpifpaf-resnet50-dynamic_b32_2.onnx \
    --fp16 \
    --workspace=16000

Also, in trtexec command, what is a difference between batch and the batch inside a shape?

3

Hi,

Please refer to the following document, which may help you:

Thank you.