Hello,
I am using the Triton server ensemble, which involves two main steps: preprocessing using .so file(s) (libtriton_.so) and executing an ONNX model compiled to a TensorRT model plan.
Currently, both the custom backend and the model in the ensemble processes one input at a time (batch size = 1). My goal is to create a version that can handle multiple inputs in parallel (batched version).
According to the Triton documentation, the server performs batching on incoming requests. What is the correct way to approach the batching of preprocessing?