Tensorrt inference on multiple batches

spolisetty · October 13, 2022, 11:19am

Hi,

Looks like your model has static inputs and the also script has some mistakes.

We recommend you to please generate an ONNX model with a dynamic input shape.
And please refer to the following sample and doc to run inference.

github.com

NVIDIA/TensorRT/blob/main/samples/python/efficientdet/build_engine.py#L144


      
          
          
    trt.init_libnvinfer_plugins(self.trt_logger, namespace="")
          
          
    self.builder = trt.Builder(self.trt_logger)
              self.config = self.builder.create_builder_config()
              self.config.max_workspace_size = workspace * (2 ** 30)
          
          
    self.network = None
              self.parser = None
          
          
def create_network(self, onnx_path, batch_size, dynamic_batch_size=None):
              """
              Parse the ONNX graph and create the corresponding TensorRT network definition.
              :param onnx_path: The path to the ONNX graph to load.
              :param batch_size: Static batch size to build the engine with.
              :param dynamic_batch_size: Dynamic batch size to build the engine with, if given,
              batch_size is ignored, pass as a comma-separated string or int list as MIN,OPT,MAX
              """
              network_flags = (1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
          
          
    self.network = self.builder.create_network(network_flags)

Thank you.

Topic		Replies	Views
Batch Inference Wrong in Python API TensorRT	15	3635	October 12, 2021
Question about Python tutorial TensorRT	3	572	October 12, 2021
Inference multiple images TensorRT TensorRT	8	2337	November 9, 2020
Tensorrt Batch Inference TensorRT tensorrt	8	1648	December 1, 2020
TensorRT Engine batch inference only has one result TensorRT tensorrt	9	1209	October 12, 2021
Tensorrt inference with batch > 1 TensorRT	4	1439	October 13, 2022
Output of batch inference TensorRT	6	2779	January 7, 2021
TensorRT Batch Inferences : empty outputs TensorRT tensorrt , jetson-inference	8	2006	July 18, 2024
Work with batch in TensorRT TensorRT tensorrt , opencv , cuda , tensorflow	20	3959	July 20, 2021
[TensorRT] C++ batch inference gives weird results TensorRT	3	1075	October 12, 2021

Tensorrt inference on multiple batches

Related topics