The input size is declared as a range of values-
“input”:
{
“min” : [1,3,320, 256],
“opt” : [1,3,320, 256],
“max” : [1,3,746, 624]
}
As part of the execution preparations, the context of the engine contains a static output shape
By using get_binding_shape we receive the same static shape, even if we change the input-
So my question is why TRT engine file make the outputs to be with static shapes:
From trt_log.txt:
“Outputs”: [
{
“Name”: “scores”,
“Location”: “Device”,
“Dimensions”: [8000,1],
“Format/Datatype”: “N/A due to dynamic shapes”
},
{
“Name”: “descs”,
“Location”: “Device”,
“Dimensions”: [1,8000,128],
“Format/Datatype”: “N/A due to dynamic shapes”
},
{
“Name”: “kpts”,
“Location”: “Device”,
“Dimensions”: [1,8000,2],
“Format/Datatype”: “N/A due to dynamic shapes”
}],
Maybe it comes from the optimization profile range min, opt & max shapes that I declared for the TRT engine process?
Please help me understand what I’m missing here,
For this specific model and specific min/opt/max shapes, is it expected that the output is dynamic? The batch size is static (=1), and only H and W are dynamic.
And we found that the model has Flatten and TopK which flattens the tensors and selects only a fixed number of top values. So, it may be expected that the number of predictions is static regardless of dynamic H and W.
So even if i expect a dynamic size of topK operator’s output(choosing k by using min between two values), so the trt would turn this operator to a fixed number?