Description
Hi,
I am running into an issue with trying the detectron2 Tensorrt sample as mentioned in https://github.com/NVIDIA/TensorRT/tree/release/8.6/samples/python/detectron2
These provided python samples for the onnx model conversion, TensorRT engine building and optimized engine inference works normally as expected using the TensorRT Python API. I am able to run inference and visualize the results on images.
However, I am writing a C++ inference app that uses these generated engine or onnx files. When I try to load the same .trt engine or parse the .onnx file using the TensorRT C++ API, I am running into plugin errors for EfficientNMS_TRT (which didn’t show up when the same engine was loaded via the python script). Below snippet shows an example of the error I am seeing:
[TRT] [W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TRT] [I] No importer registered for op: EfficientNMS_TRT. Attempting to import as plugin.
[TRT] [I] Searching for plugin: EfficientNMS_TRT, plugin_version: 1, plugin_namespace:
[TRT] [E] 3: getPluginCreator could not find plugin: EfficientNMS_TRT version: 1
[TRT] [E] ModelImporter.cpp:771: While parsing node number 254 [EfficientNMS_TRT -> "num_detections_rpn"]:
[TRT] [E] ModelImporter.cpp:772: --- Begin node ---
[TRT] [E] ModelImporter.cpp:773: input: "anchors_3"
input: "scores_unsqueeze:0_2"
input: "default_anchors"
output: "num_detections_rpn"
output: "detection_boxes_rpn"
output: "detection_scores_rpn"
output: "detection_classes_rpn"
name: "nms_rpn"
op_type: "EfficientNMS_TRT"
attribute {
name: "plugin_version"
s: "1"
type: STRING
}
attribute {
name: "background_class"
i: -1
type: INT
}
attribute {
name: "max_output_boxes"
i: 1000
type: INT
}
attribute {
name: "score_threshold"
f: 0.01
type: FLOAT
}
attribute {
name: "iou_threshold"
f: 0.7
type: FLOAT
}
attribute {
name: "score_activation"
i: 0
type: INT
}
attribute {
name: "class_agnostic"
i: 0
type: INT
}
attribute {
name: "box_coding"
i: 1
type: INT
}
[TRT] [E] ModelImporter.cpp:774: --- End node ---
[TRT] [E] ModelImporter.cpp:777: ERROR: builtin_op_importers.cpp:5404 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
What is surprising is that same plugin (EfficientNMS_TRT) works fine with the TensorRT python API but causes this error with the C++ API. I am not sure why its not able to link/find that plugin with C++ API.
I ensured I am calling the initLibNvInferPlugins(&trt_logger_, ""); to register various plugins. I print some debug info of the registered plugins after this function call using this code snippet:
// Temporarily print the registered plugins
int numCreators = 0;
nvinfer1::IPluginCreator* const* tmpList = getPluginRegistry()->getPluginCreatorList(&numCreators);
for (int k = 0; k < numCreators; ++k)
{
if (!tmpList[k])
{
std::cout << "Plugin Creator for plugin " << k << " is a nullptr." << std::endl;
continue;
}
std::string pluginName = tmpList[k]->getPluginName();
std::cout << k << ": " << pluginName << std::endl;
}
and I do see the EfficientNMS_TRT being printed as seen below from the console output
0: RNNTEncoderPlugin
1: SmallTileGEMM_TRT
2: DLRM_BOTTOM_MLP_TRT
3: CustomQKVToContextPluginDynamic
4: CustomQKVToContextPluginDynamic
5: CustomQKVToContextPluginDynamic
6: CustomSkipLayerNormPluginDynamic
7: CustomSkipLayerNormPluginDynamic
8: CustomSkipLayerNormPluginDynamic
9: CustomSkipLayerNormPluginDynamic
10: SingleStepLSTMPlugin
11: RnRes2FullFusion_TRT
12: RnRes2Br2bBr2c_TRT
13: RnRes2Br2bBr2c_TRT
14: RnRes2Br1Br2c_TRT
15: RnRes2Br1Br2c_TRT
16: GroupNormalizationPlugin
17: CustomGeluPluginDynamic
18: CustomFCPluginDynamic
19: CustomEmbLayerNormPluginDynamic
20: CustomEmbLayerNormPluginDynamic
21: CustomEmbLayerNormPluginDynamic
22: DisentangledAttention_TRT
23: BatchedNMSDynamic_TRT
24: BatchedNMS_TRT
25: BatchTilePlugin_TRT
26: Clip_TRT
27: CoordConvAC
28: CropAndResizeDynamic
29: CropAndResize
30: DecodeBbox3DPlugin
31: DetectionLayer_TRT
32: EfficientNMS_Explicit_TF_TRT
33: EfficientNMS_Implicit_TF_TRT
34: EfficientNMS_ONNX_TRT
35: EfficientNMS_TRT
36: FlattenConcat_TRT
37: GenerateDetection_TRT
38: GridAnchor_TRT
39: GridAnchorRect_TRT
40: InstanceNormalization_TRT
41: InstanceNormalization_TRT
42: LReLU_TRT
43: ModulatedDeformConv2d
44: MultilevelCropAndResize_TRT
45: MultilevelProposeROI_TRT
46: MultiscaleDeformableAttnPlugin_TRT
47: NMSDynamic_TRT
48: NMS_TRT
49: Normalize_TRT
50: PillarScatterPlugin
51: PriorBox_TRT
52: ProposalDynamic
53: ProposalLayer_TRT
54: Proposal
55: PyramidROIAlign_TRT
56: Region_TRT
57: Reorg_TRT
58: ResizeNearest_TRT
59: ROIAlign_TRT
60: RPROI_TRT
61: ScatterND
62: SpecialSlice_TRT
63: Split
64: VoxelGeneratorPlugin
Please provide any suggestions or fixes to this as I have tried searching through similar issues on this forum but have not had success with any of the solutions. Thank you.
Environment
TensorRT Version: 8.6.1 GA
GPU Type: RTX 3050 Ti
Nvidia Driver Version: 530.30.02
CUDA Version: 11.7
CUDNN Version: 8.8.0.121
Operating System + Version: Ubuntu 22.04
Python Version (if applicable): 3.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.13.1
Baremetal or Container (if container which image + tag):