Help to deal with IBuilder, INetworkDefinition, IBuilderConfig and other

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

What are they needed for IBuilder, INetworkDefinition, IBuilderConfig, IOptimizationProfile, IRuntime, ICudaEngine, IExecutionContext?

What does each of them do and why is each of them needed? In what order should I create what? I am primarily interested in running TensorRT engine with dynamic shapes and dynamic batch_size.

I will be grateful for the answer,
Egor

2

The above objects in the TensorRT API are used to build, optimize, and execute a TensorRT engine.

  • IBuilder The top-level object used to create and customize the other objects. It allows you to set the maximum batch size, the precision of the engine, and other parameters.
  • INetworkDefinition is a representation of the neural network graph. It contains the nodes and tensors that make up the network.
  • IBuilderConfig is used to configure the optimization process for the engine. It allows you to specify things like the number of optimization passes, the level of precision, and the hardware accelerator to use.
  • IOptimizationProfile is used to specify the performance characteristics of the engine for different input sizes. This can be useful for networks that are used for inference with different batch sizes.
  • IRuntime is used to load and execute the TensorRT engine.
  • ICudaEngine is the actual TensorRT engine. It is a compiled version of the neural network graph that can be executed on the GPU.
  • IExecutionContext is used to execute the TensorRT engine. It provides methods for setting the input and output tensors and for running the inference.

Please refer to the TensorRT developer guide and samples for more details.

Thank you.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.