PluginV2Layer must be V2DynamicExt when there are runtime input dimensions

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Hi Nvidia Team,

Actually, I have a custom model built from Pytorch, I have converted .pt to .onnx, but there is one Op which is not supported in TensorRT(Einsum).

I have Implemented the custom Plugin for that unsupported Op, but while converting from ONNX to TRT I am facing an error as: Einsum_194: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions
I am not sure where I did wrong.

The command used is: ./trtexec --onnx=/home/darshan/slow_fast.onnx --explicitBatch --shapes=input1:1x3x8x256x455,input2:1x3x32x256x455 --plugins=/home/darshan/Einsum_op.so

The image of the error has been attached below:

Screenshot from 2021-02-22 01-04-57
Screenshot from 2021-02-22 01-04-571835×281 81 KB

Can you please help me to resolve this error? Where Exactly I am going wrong?what’s the solution for this?

Thanks and Regards,
Darshan C G

2

Hi, Request you to check the below reference links for custom plugin implementation.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleOnnxMnistCoordConvAC

Thanks!

3

Hello @NVES,

Thanks for your reply.

I have already implemented the Custom Plugin. I wanted to know the reason for the
error I am getting.

Thanks and Regards,
Darshan

4

Hi @darshancganji12,

Could you please validate your onnx model before converting to trt.
with the below snippet
check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).

Please share us verbose logs as well for better debugging.

Hope this link helps you.

Thank you.

5

Hi @spolisetty,

I am sharing the verbose of trtexec:

        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Conv_184 for ONNX node: Conv_184
        [02/22/2021-12:45:06] [V] [TRT] builtin_op_importers.cpp:533: Using kernel: (1, 1, 1), strides: (1, 1, 1), prepadding: (0, 0, 0), postpadding: (0, 0, 0), dilations: (1, 1, 1), numOutputs: 512
        [02/22/2021-12:45:06] [V] [TRT] builtin_op_importers.cpp:534: Convolution output dimensions: (1, 512, 4, 8, 14)
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1572 for ONNX tensor: 1572
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Conv_184 [Conv] outputs: [1572 -> (1, 512, 4, 8, 14)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze_185 [Unsqueeze]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1559
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Unsqueeze_185 [Unsqueeze] inputs: [1559 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] onnx2trt_utils.cpp:1793: Original shape: (), unsqueezing to: (1,)
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Unsqueeze_185 for ONNX node: Unsqueeze_185
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1575 for ONNX tensor: 1575
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Unsqueeze_185 [Unsqueeze] outputs: [1575 -> (1)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Concat_186 [Concat]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1575
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 2878
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 2879
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Concat_186 [Concat] inputs: [1575 -> (1)], [2878 -> (1)], [2879 -> (1)], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Concat_186 for ONNX node: Concat_186
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1578 for ONNX tensor: 1578
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Concat_186 [Concat] outputs: [1578 -> (3)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Reshape_187 [Reshape]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1569
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1578
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Reshape_187 [Reshape] inputs: [1569 -> (1, 512, 8, 16, 29)], [1578 -> (3)], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Reshape_187 for ONNX node: Reshape_187
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1579 for ONNX tensor: 1579
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Reshape_187 [Reshape] outputs: [1579 -> (-1, -1, -1)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze_188 [Unsqueeze]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1559
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Unsqueeze_188 [Unsqueeze] inputs: [1559 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] onnx2trt_utils.cpp:1793: Original shape: (), unsqueezing to: (1,)
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Unsqueeze_188 for ONNX node: Unsqueeze_188
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1582 for ONNX tensor: 1582
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Unsqueeze_188 [Unsqueeze] outputs: [1582 -> (1)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Concat_189 [Concat]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1582
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 2880
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 2881
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Concat_189 [Concat] inputs: [1582 -> (1)], [2880 -> (1)], [2881 -> (1)], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Concat_189 for ONNX node: Concat_189
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1585 for ONNX tensor: 1585
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Concat_189 [Concat] outputs: [1585 -> (3)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Reshape_190 [Reshape]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1571
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1585
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Reshape_190 [Reshape] inputs: [1571 -> (1, 512, 4, 8, 14)], [1585 -> (3)], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Reshape_190 for ONNX node: Reshape_190
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1586 for ONNX tensor: 1586
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Reshape_190 [Reshape] outputs: [1586 -> (-1, -1, -1)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze_191 [Unsqueeze]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1559
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Unsqueeze_191 [Unsqueeze] inputs: [1559 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] onnx2trt_utils.cpp:1793: Original shape: (), unsqueezing to: (1,)
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Unsqueeze_191 for ONNX node: Unsqueeze_191
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1589 for ONNX tensor: 1589
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Unsqueeze_191 [Unsqueeze] outputs: [1589 -> (1)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Concat_192 [Concat]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1589
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 2882
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 2883
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Concat_192 [Concat] inputs: [1589 -> (1)], [2882 -> (1)], [2883 -> (1)], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Concat_192 for ONNX node: Concat_192
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1592 for ONNX tensor: 1592
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Concat_192 [Concat] outputs: [1592 -> (3)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Reshape_193 [Reshape]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1572
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1592
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Reshape_193 [Reshape] inputs: [1572 -> (1, 512, 4, 8, 14)], [1592 -> (3)], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Reshape_193 for ONNX node: Reshape_193
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1593 for ONNX tensor: 1593
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Reshape_193 [Reshape] outputs: [1593 -> (-1, -1, -1)], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Einsum_194 [Einsum]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1579
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1586
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Einsum_194 [Einsum] inputs: [1579 -> (-1, -1, -1)], [1586 -> (-1, -1, -1)], 
        [02/22/2021-12:45:06] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: Einsum. Attempting to import as plugin.
        [02/22/2021-12:45:06] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: Einsum, plugin_version: 1, plugin_namespace: 
        [02/22/2021-12:45:06] [I] [TRT] builtin_op_importers.cpp:3676: Successfully created plugin: Einsum
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Einsum_194 for ONNX node: Einsum_194
        [02/22/2021-12:45:06] [E] [TRT] Einsum_194: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1594 for ONNX tensor: 1594
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Einsum_194 [Einsum] outputs: [1594 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Shape_195 [Shape]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1594
        [02/22/2021-12:45:06] [E] [TRT] Einsum_194: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Shape_195 [Shape] inputs: [1594 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:141: Registering layer: Shape_195 for ONNX node: Shape_195
        [02/22/2021-12:45:06] [E] [TRT] Einsum_194: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
        [02/22/2021-12:45:06] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 1595 for ONNX tensor: 1595
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Shape_195 [Shape] outputs: [1595 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Constant_196 [Constant]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Constant_196 [Constant] inputs: 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:179: Constant_196 [Constant] outputs: [1596 -> ()], 
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:103: Parsing node: Gather_197 [Gather]
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1595
        [02/22/2021-12:45:06] [E] [TRT] Einsum_194: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:119: Searching for input: 1596
        [02/22/2021-12:45:06] [V] [TRT] ModelImporter.cpp:125: Gather_197 [Gather] inputs: [1595 -> ()], [1596 -> ()], 
        [02/22/2021-12:45:06] [E] [TRT] Einsum_194: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
        ERROR: onnx2trt_utils.cpp:188 In function convertAxis:
        [8] Assertion failed: axis >= 0 && axis < nbDims
        [02/22/2021-12:45:06] [E] Failed to parse onnx file
        [02/22/2021-12:45:06] [E] Parsing model failed
        [02/22/2021-12:45:06] [E] Engine creation failed
        [02/22/2021-12:45:06] [E] Engine set up failed
        &&&& FAILED TensorRT.trtexec # ./trtexec --onnx=/home/darshan/slow_fast.onnx --shapes=input1:1x3x8x256x455,input2:1x3x32x256x455 --plugins=/home/darshan/Einsum_op.so --verbose

I have implemented the plugin for Einsum Operator.
Below is the node Information:

Screenshot from 2021-02-22 14-06-40
Screenshot from 2021-02-22 14-06-401920×1080 185 KB

If you can observe in the verbose:
Reshape_187 [Reshape] inputs: [1569 → (1, 512, 8, 16, 29)], [1578 → (3)], outputs: [1579 → (-1, -1, -1)]
Reshape_190 [Reshape] inputs: [1571 → (1, 512, 4, 8, 14)], [1585 → (3)], outputs: [1586 → (-1, -1, -1)]

Reshape_187 and Reshape_190 are going as an input to the einsum_194. But the Output shape info of both the reshape is (-1, -1, -1)(which is dynamic), so may be the reason I have to implement the Plugin for Dynamic case.
Is it right? Or is there any other way?

Thanks,
Darshan

6

Hi @spolisetty and @NVES,

I have solved this Issue by Inheriting IPluginv2DynamicExt Class instead of IPluginv2Ext Class.

Thanks

1 Like