How to combine TRTIS dynamic batching with TRT engine of dynamic batchsize?

AI & Data Science Deep Learning (Training & Inference) Triton Inference Server - archived
1

Hi,

We have a TRT engine which is parsed from a onnx model. Then TRT engine have a dynamic batchsize, which is generated by:
trtexec --explicitBatch --onnx=alexnet_dynamic.onnx
–minShapes=actual_input_1:1x3x224x224 \ # kMIN shape
–optShapes=actual_input_1:3x3x224x224 \ # kOPT shape
–maxShapes=actual_input_1:5x3x224x224 \ # kMAX shape
–shapes=actual_input_1:3x3x224x224 \ # Inference shape - this is like context->setBindingShape(3,3,224,224)
–saveEngine=alexnet_dynamic.engine

And every time the input batch changes, I have to set the size of input bindings before execute inference:

image
image2337×819 79.2 KB

So how could we combine TRTIS dynamic batching with TRT engine of dynamic batchsize? Will TRTIS set the input shape to TRT engine when the actual input shape changed?