I have a problem to register a pluginV2 to the Onnx parser as opposed to the uff parser which successfully can be registered to it.
This is my configuration:
Pyton - 3.6.8
Tensorflow Python\C++ (TF)- 1.9 (C++ version was built from sources)
TensorRT C++ (TRT) - 126.96.36.199
CuDNN - 7.6.3
CUDA - 9.0
I have two models:
- YoloV3 - Implemeted and trained via TF Python, Intended to be inferenced via TRT C++
- SegNet- Implemeted and trained via PyTorch, Intended to be inferenced via TRT C++
YoloV3 model includes one unsupported TF Upsample command which implemented as a TRT plugin by two ways (for good practice, in actually only the first method is used):
- nvinfer1::IPluginCreator, nvinfer1::IPluginV2 and REGISTER_TENSORRT_PLUGIN
- nvinfer1::IPluginFactory, public nvuffparser::IPluginFactoryExt, nvinfer1::IPluginExt and nvuffparser::IUffParser setPluginFactoryExt service
In both ways the PlugIn registry to the uff parser is working well.
I also edited the model with the graphsurgeon in order to replace the upsample name and remove some additional unnecessary nodes.
Additionally, just to make sure that it isn’t must to use graphsurgeon in order to declare on a layer as a plugin layer, I intentionally declared an additional pluging layer without any graphsurgeon manipulation.
I used the first method to do that and it works well, my plugin creator call (inherited from the nvinfer1::IPluginCreator) constructor and createPlugin methods are activated during the nvuffparser::IUffParser parse operation.
The PlugIn creator can be identified by the parser and its methods activated during the parsering process and the CUDA engine build process and finally inference is working well as expected with the plugin methods (enqueue …).
Now, I’m trying to do the same thing for the SegNet model.
The segNet PyTorch was converted to an Onnx format using the following commands
dummy_input = torch.randn(1, 32, 400, 400, device='cuda') input_names = ["Input"] output_names = ["Output"] torch.onnx.export(model, dummy_input, "segNet.onnx", verbose=True, input_names=input_names, output_names=output_names)
At the beginning the command torch.onnx.export was failed due to unsupported max_unpool2d command.
I updated the file symbolic.py inside the onnx directory of the torch package:
def max_unpool2d(g, self, indices, output_size): return g.op("max_unpool2d", self, indices, output_size)
After this update the operation torch.onnx.export starts to work properly without any errors and a segNet.onnx file was successfully generated.
Now I already know that I will need to implement a plugin for the unsupported max_unpool2d PyTorch operation (as I did for the upsample TF operation described above).
Also when I activated this operation:
auto parsed = m_onnxParser->parseFromFile( fileName.string().c_str(), static_cast<int>(nvinfer1::ILogger::Severity::kINFO));
I got the following report:
Input filename: …/…/…/…/Data/SegNet/Semantic_Segmentation/segNet.onnx
ONNX IR version: 0.0.4
Opset version: 9
Producer name: pytorch
Producer version: 1.1
Model version: 0
WARNING: ONNX model has a newer ir_version (0.0.4) than this parser was built against (0.0.3).
……. Some good reports…….
WARNING: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Successfully casted down to INT32.
While parsing node number 8 [Gather]:
ERROR: onnx2trt_utils.hpp:347 In function convert_axis:
 Assertion failed: axis >= 0 && axis < nbDims
Error code - 8
Error description - Assertion failed: axis >= 0 && axis < nbDims
Error source file - onnx2trt_utils.hpp
Error source line - 347
Error function - convert_axis
Error node – 8
I checked the Onnx parser code from the TensorRT OSS package and I realized that I need to implement my Gather operation as a plugin too.
I implemented the Gather plugin exactly as I implemented for the upsample plugin above, based on the first method (described above) and when I started to run, the constructor of my Gather plugin creator (inherited from the nvinfer1::IPluginCreator) was activated due to the REGISTER_TENSORRT_PLUGIN macro but the createPlugin method wasn’t activated during the nvonnxparser::IParser parseFromFile operation and still I got the same error report (described above)
Please advise how plugin shall be registered to the onnx parser?