How to set TensorRT context profilers in order to support more than one dynamic input?

AI & Data Science Deep Learning (Training & Inference) TensorRT
#1

Description

We have a model with two tensors inputs.

Each tensor input has its own optional dimensions sizes.

  • Tensor#0 has two options:
    Dimenstions#0 - {N = 1, C = 1, H = 320, W = 640}
    Dimenstions#1 - {N =1, C = 1, H = 256, W =512}
    Each element type is fp 32 bits

  • Tensor#1 has three options:
    Dimenstions#0 - { N =1, C = 1024}
    Dimenstions#1 - { N =1, C = 512}
    Dimenstions#2 - { N =1, C = 800}
    Each element type is Int 32 bits

I think that each tensor shall has its own profile range:

  • Profile#0:
    Min - Dims4(1, 1, 256, 512)
    Max - Dims4(1, 1, 320, 640)
    Opt - Dims4(1, 1,300, 568) //Our most commonly used runtime values

  • Profile#1:
    Min - Dims4(1, 512)
    Max - Dims4(1, 1024)
    Opt - Dims4(1, 768) //Our most commonly used runtime values

I’m adding two profiles to my builder config using the method addOptimizationProfile not before I’m building them using the setDimensions method to declare their kMIN, kOPT and kMAX.

I understand that I shall set the actual dimensions of each tensor input using the method setBindingDimensions of the context before I activate its enqueueV2 method.

These are my questions:

I cannot understand the relations between the tensors and profiles?

Shall I move between the two profiles in order to set the two tensors inputs actual dimensions? If so, how shall I do it?
Or
Since in setOptimizationProfile API description it is mentioned
“Select an optimization profile for the current context…” ,
shall I understand that one profile shall reply all tensor inputs dimensions??

Can you provide a code example how to select profile, set the actual tensor input dimension and then activate the inference process?

Environment

TensorRT Version: 7.2.3.4
GPU Type: Quadro M2000M
Nvidia Driver Version: R451.77
CUDA Version: 11.0
CUDNN Version: 8.0.1
Operating System + Version: Microsoft WIndows 10 Enterprise 2016
Python Version (if applicable): 3.6.8
TensorFlow Version (if applicable): NA
PyTorch Version (if applicable): NA
Baremetal or Container (if container which image + tag): Baremetal

Relevant Files

NA

Steps To Reproduce

Suppose I want Dimenstions#0 for tensor input#0 and Dimenstions#2 for tensor input#1,shall I do it in this way (pseudo code)?

context->setOptimizationProfile(Profile#0);
context->setBindingDimensions(Tensor#0, Dimenstions#0);

context->setOptimizationProfile(Profile#1);
context->setBindingDimensions(Tensor#1, Dimenstions#2);

if true == context->allInputDimensionsSpecified()
context->enqueueV2((void**)&memory, *stream, nullptr))

If not, please provide an example

Thanks,

#5

Hi @orong13,

Please refer following, which may help you.
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work_dynamic_shapes

Thank you.

#6

Thanks @spolisetty,

I know this link and as you said based on its information I successfully inference my model which includes only one tensor input with dynamic shape.

My tensor input has two sizes options so I built a profile which its kMIN & KMAX enable both sizes to work properly.
In this simple scenario I have only one profile which its index is 0 and selected using the method setOptimizationProfile before I’m setting the actual dimensions I need to use in current enqueueV2 activation.

But, how and what shall I declare in case I have more than one tensor input with dynamic shape?
These two tensor inputs are not related and each one has its own optional dimensions values ranges.
Shall I declare two different profiles? Each one for each tensor input?
Shall I set the relevant profile index and then set its tensor input dimensions for both profiles and inputs (Before enqueue activation)? (Please see above my pseudo code)

How TRT engine\context know which profile is relevant when tensor input dimensions settings is activated? Does it know how to toggle between all profiles?

If I didn’t miss this scenario is not detailed in the link…

Thanks,

#7

@orong13,

Optimization profiles can define shape ranges for multiple inputs. So we can create a single profile with separate min/opt/max values for each input tensor.
TensorRT cannot toggle between profiles by itself. Profile has to be chosen by user using setOptimizationProfile.

Thank you.

#8

Ok @spolisetty, I think now it is clear.
I also made a test case which help me to better understand how to handle multi inputs with dynamic shapes.

Now I understand that I can have several inputs, each one has its own range of shape dimensions and I shall set the actual one of each one of them before I activate the enqueuv2 method.

Now I cannot find any real reason why or when I will need to declare more than one profile.

Can you provide me please a real scenario\model reference which require more than one profile and I will further learn about it?

Thanks,

#9

Hi @orong13,

The previous response was with respect to your query of having more than one tensor input.
However you may find the below link useful
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#opt_profiles

Thank you.

#10

like this demo: Optimization profile with dynamic batches doesn't behave the same as setMaxBatchSize · Issue #370 · NVIDIA/TensorRT · GitHub