Dynamic onnx model becomes a trt engine with a static output

Description

A fully dynamic onnx model is converted to a tensortrt engine, but for an unknown reason the engine’s output becomes static for different input sizes.

Environment

TensorRT Version: 8.4.1.5
GPU Type: Quadro T2000
Nvidia Driver Version: R471.68 (r471_59-5) / 30.0.14.7168 (8-5-2021)
CUDA Version: 11.4
CUDNN Version: 8.1.1
Operating System + Version: Windows 10
Python Version (if applicable): 3.6.8
TensorFlow Version (if applicable): NA
PyTorch Version (if applicable): 1.10.0
Baremetal or Container (if container which image + tag): Baremetal

Relevant Files

model_folded.onnx (2.8 MB)

Steps To Reproduce

The input size is declared as a range of values-
“input”:
{
“min” : [1,3,320, 256],
“opt” : [1,3,320, 256],
“max” : [1,3,746, 624]
}

As part of the execution preparations, the context of the engine contains a static output shape
By using get_binding_shape we receive the same static shape, even if we change the input-

bindingShape = self.context.get_binding_shape(binding=bindingIdx)

For some reason the trt optimization changes the output size to a static shape, without any connection to the input.

here is the trt log-
trt_log.txt (826.7 KB)

Hi,

Please refer to the following similar issue.
TRT Polygraphy change a dynamic tensor output to static output,

Thank you.

Hello,
I don’t think this is the same case.
In the referenced case, you are right that the input model has static input shape and it was my mistake:

But here the iput model has dynamic input shape:

It also can be seen in the attached trt_log.txt which report on dynamic input (line 103):

Adding network input: input with dtype: float32, dimensions: (-1, -1, -1, -1)

So my question is why TRT engine file make the outputs to be with static shapes:
From trt_log.txt:

“Outputs”: [
{
“Name”: “scores”,
“Location”: “Device”,
“Dimensions”: [8000,1],
“Format/Datatype”: “N/A due to dynamic shapes”
},
{
“Name”: “descs”,
“Location”: “Device”,
“Dimensions”: [1,8000,128],
“Format/Datatype”: “N/A due to dynamic shapes”
},
{
“Name”: “kpts”,
“Location”: “Device”,
“Dimensions”: [1,8000,2],
“Format/Datatype”: “N/A due to dynamic shapes”
}],

Maybe it comes from the optimization profile range min, opt & max shapes that I declared for the TRT engine process?
Please help me understand what I’m missing here,

Thanks

Hi,

Could you please share with us the original ONNX model(before running polygraphy) for better debugging?

Thank you.

That’s the original ONNX-
model.onnx (3.3 MB)

Thank you for your support

Hi,

For this specific model and specific min/opt/max shapes, is it expected that the output is dynamic? The batch size is static (=1), and only H and W are dynamic.

And we found that the model has Flatten and TopK which flattens the tensors and selects only a fixed number of top values. So, it may be expected that the number of predictions is static regardless of dynamic H and W.

Thank you.

So even if i expect a dynamic size of topK operator’s output(choosing k by using min between two values), so the trt would turn this operator to a fixed number?

Hi,

Currently dynamic K is not supported.

Thank you.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.