How to enable dynamic batching for models on triton inference server


I am trying to host a custom model on Triton Inference server and I am trying to enable dynamic batching for the model.

I am converting a pytorch model to onnx and enabling dynamic batching on the input and output nodes.

I attach a copy of the script i use to convert from pytorch to onnx in the attachments and a copy of my config.pbtxt file.

Relevant Files (683 Bytes)
config.pbtxt (270 Bytes)

Error Encountered

E0522 09:23:57.598550 82] failed to load ‘par’ version 1: Invalid argument: model ‘par’, tensor ‘512’: for the model to support batching the shape should have at least 1 dimension and the first dimension must be -1; but shape expected by the model is [1,22]