max batch size is missing in configurePlugin of IPluginV2IOExt.

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

some plugin needs max batch size to allocate cpu buffers in initialize, but it is missing in configurePlugin of IPluginV2IOExt.

2

Hi,

Max Batch Size is a property of implicit networks, IPluginV2IOExt and IPluginV2DynamicExt are for full dims or dynamic shape networks.

If you want the same functionality as implicit networks, create a network with -1 in the first dimension and then create a profile with kMAX as the max batch size in that dimension.

Thanks

3

I just need fixed batch size, but I can’t get batch dim in configurePlugin of IPluginV2IOExt, the dims of the PluginTensorDesc don’t contain batch dim.

Is there a way to get explicit batch size of a tensor before enqueue when using IPluginV2IOExt?

4

Hi,

You can save explicit batch size (or even the dims of all I/O tensors) when you implement configurePlugin to members, and serialize it via void serialize(void* buffer) const.
When you do inference at runtime stage, that serialized batch will be deserialized via IPluginV2* IPluginCreator::deserializePlugin(const char* name, …).

Thanks

5

The problem is that arguments of the configurePlugin don’t contain batch size even if I use explicit batch axis network:
[b]

virtual void nvinfer1::IPluginV2IOExt::configurePlugin ( const PluginTensorDesc * in,
int nbInput,
const PluginTensorDesc * out,
int nbOutput
)
[/b]
My workaround is use second axis as batch axis.