Plugin in TensorRT and dynamic batch inference

Now I can change dynamic batch size in inference for TensorRT.

Batch size can be varied between min to max according to optimization profile.

But the issue is at plugin.

Even though, I use IPluginV2DynamicExt. Input/Output dimension are fixed.
For example, plugin input dimension is batchsize(lets say 10) x 20 x 40.
batchsize is fixed as 10 in plugin creation time.
When I change batch size at input of network dynamically (i.e. every iteration of inference, batch size is changed), the plugin has fixed input size as 10x20x40 and fixed output size, always 10.
It has never changed with different batch size.
Is it possible to change the plugin’s batch size dynamically?

That means network’s batchsize is dynamically changed but plugin’s batch size is fixed.
I will have memory issue after running some time.

Hi @edit_or,
Request you to provide your script, as in an ideal scenario this should not happen.

Thanks!

Thanks. My script is attached.Recognition.cpp (34.4 KB)

Hi @edit_or,

auto profile = builder->createOptimizationProfile();
profile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(10, 24, 94, 3));
profile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(10, 24, 94, 3));
profile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(10, 24, 94, 3));
config->addOptimizationProfile(profile);

The profile setup is incorrect.
Instead, Use (1, 24, 94, 3) as min profile.
Use same value for min and max will generate a static engine.

Thanks!

I have this error when I set 1 to min.
input:0: kMIN dimensions in profile 0 are [1,24,94,3] but input has static dimensions [10,24,94,3]. Your test didn’t have this error?

Hi @edit_or,
You may need to adjust your model.
It should be dynamic.

Thanks!

I don’t understand. What to adjust? I set onnx model dimensions as -1,24,94,3. Then your tensorrt does not accept -1. I have error. I have already discussed in earlier discussion. Your tensorrt doesn’t accept -1 input. To be dynamic, if you don’t accept -1, what should I put? I can’t find now. In earlier discussion, dimension needs to be positive. Positive dimension is fixed dimension.