RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

The version of TensorRT is 10.3.0.

Hi,

About performance, have you tried the below command to maximize performance first?

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

Based on the log you shared above, it looks like there are some non-supported models inside your model.

How do you convert the model into ONNX?
Do you export it with PyTorch directly?

Thanks.

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

When I tried this command, the FPS of Mask R-CNN inference on real-time camera footage went from 1.0 to about 2.5.
However, I would like to increase the FPS even further, so I would like to try the TensorRT engine.

When converting to an ONNX model, I ran the following code.

from mmdeploy.apis import torch2onnx
import torch

入力画像(ダミーデータとして使用)

dummy_img = ‘/home/andolab/mmdet_env/demo.jpg’ # 実際の画像パス、またはダミーとして使用する画像ファイル

作業ディレクトリ

work_dir = ‘/home/andolab/mmdet_env’

ONNX出力ファイル名

save_file = ‘/home/andolab/mmdet_env/model.onnx’

MMDeploy用のデプロイ設定ファイル

deploy_cfg = ‘/home/andolab/mmdet_env/mmdeploy/configs/mmdet/instance-seg/instance-seg_tensorrt_dynamic-320x320-1344x1344.py’

モデル設定ファイル

model_cfg = ‘/home/andolab/mmdet_env/configs/mask_rcnn/my_config.py’

学習済みモデルのチェックポイント

model_checkpoint = ‘/home/andolab/mmdet_env/epoch_13.pth’

GPUデバイスを指定

device = ‘cuda:0’

ONNX形式に変換

torch2onnx(
img=dummy_img,
work_dir=work_dir,
save_file=save_file,
deploy_cfg=deploy_cfg,
model_cfg=model_cfg,
model_checkpoint=model_checkpoint,
device=device
)

print(f"ONNXモデルが {save_file} に保存されました。")

This is MMdeploy, which supports MMDetection.
I was able to convert to an ONNX model successfully, but when I try to convert to TensorRT-fp16 with this code, I get this error.

[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:951: — End node —
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:953: ERROR: onnxOpCheckers.cpp:780 In function checkFallbackPluginImporter:
[6] creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:948: While parsing node number 1030 [TRTBatchedNMS → “/TRTBatchedNMS_1_output_0”]:
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:950: — Begin node —
input: “/Unsqueeze_116_output_0”
input: “/Slice_14_output_0”
output: “/TRTBatchedNMS_1_output_0”
output: “/TRTBatchedNMS_1_output_1”
name: “/TRTBatchedNMS_1”
op_type: “TRTBatchedNMS”
attribute {
name: “background_label_id”
i: -1
type: INT
}
attribute {
name: “clip_boxes”
i: 0
type: INT
}
attribute {
name: “iou_threshold”
f: 0.5
type: FLOAT
}
attribute {
name: “is_normalized”
i: 0
type: INT
}
attribute {
name: “keep_topk”
i: 100
type: INT
}
attribute {
name: “num_classes”
i: 6
type: INT
}
attribute {
name: “return_index”
i: 0
type: INT
}
attribute {
name: “score_threshold”
f: 0.05
type: FLOAT
}
attribute {
name: “topk”
i: 5000
type: INT
}
domain: “mmdeploy”

[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:951: — End node —
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:953: ERROR: onnxOpCheckers.cpp:780 In function checkFallbackPluginImporter:
[6] creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:948: While parsing node number 1064 [MMCVMultiLevelRoiAlign → “/mask_roi_extractor/MMCVMultiLevelRoiAlign_output_0”]:
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:950: — Begin node —
input: “/Reshape_61_output_0”
input: “/neck/fpn_convs.0/conv/Conv_output_0”
input: “/neck/fpn_convs.1/conv/Conv_output_0”
input: “/neck/fpn_convs.2/conv/Conv_output_0”
input: “/neck/fpn_convs.3/conv/Conv_output_0”
output: “/mask_roi_extractor/MMCVMultiLevelRoiAlign_output_0”
name: “/mask_roi_extractor/MMCVMultiLevelRoiAlign”
op_type: “MMCVMultiLevelRoiAlign”
attribute {
name: “aligned”
i: 1
type: INT
}
attribute {
name: “featmap_strides”
floats: 4
floats: 8
floats: 16
floats: 32
type: FLOATS
}
attribute {
name: “finest_scale”
i: 56
type: INT
}
attribute {
name: “output_height”
i: 14
type: INT
}
attribute {
name: “output_width”
i: 14
type: INT
}
attribute {
name: “pool_mode”
i: 1
type: INT
}
attribute {
name: “roi_scale_factor”
f: 1
type: FLOAT
}
attribute {
name: “sampling_ratio”
i: 0
type: INT
}
domain: “mmdeploy”

[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:951: — End node —
[01/29/2025-19:55:35] [E] [TRT] ModelImporter.cpp:953: ERROR: onnxOpCheckers.cpp:780 In function checkFallbackPluginImporter:
[6] creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[01/29/2025-19:55:35] [E] Failed to parse onnx file
[01/29/2025-19:55:35] [I] Finished parsing network model. Parse time: 0.442071
[01/29/2025-19:55:35] [E] Parsing model failed
[01/29/2025-19:55:35] [E] Failed to create engine from model or file.
[01/29/2025-19:55:35] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v100300] # trtexec --onnx=/home/andolab/model.onnx --saveEngine=/home/andolab/engine.trt --plugins=/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so --fp16

I have built the TensorRT plugin and arranged it like this.

(mmdet_env) andolab@andolab:~$ ls /usr/lib/aarch64-linux-gnu | grep libnvinfer
libnvinfer.so
libnvinfer.so.10
libnvinfer.so.10.3.0
libnvinfer_builder_resource.so.10.3.0
libnvinfer_dispatch.so
libnvinfer_dispatch.so.10
libnvinfer_dispatch.so.10.3.0
libnvinfer_dispatch_static.a
libnvinfer_lean.so
libnvinfer_lean.so.10
libnvinfer_lean.so.10.3.0
libnvinfer_lean_static.a
libnvinfer_plugin.so
libnvinfer_plugin.so.10
libnvinfer_plugin.so.10.3.0
libnvinfer_plugin_static.a
libnvinfer_static.a
libnvinfer_vc_plugin.so
libnvinfer_vc_plugin.so.10
libnvinfer_vc_plugin.so.10.3.0
libnvinfer_vc_plugin_static.a

I looked at MMdeploy’s Git and built it to support TensorRT, but it seems that IPluginV2 is deprecated and I need to change to IPluginV3, so I am currently checking these files manually.

/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:35:10: error: cannot convert ‘mmdeploy::TRTBicubicInterpolate*’ to ‘nvinfer1::IPluginV2DynamicExt*’ in return
35 | return plugin;
| ^~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.hpp:9,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp: At global scope:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:83:22: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
83 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:147:66: note: in expansion of macro ‘TRT_NOEXCEPT’
147 | const char name, const nvinfer1::PluginFieldCollection fc) TRT_NOEXCEPT {
| ^~~~~~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.hpp:9,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp: In member function ‘virtual nvinfer1::IPluginV2
mmdeploy::TRTBicubicInterpolateCreator::createPlugin(const char
, const nvinfer1::PluginFieldCollection*)’:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:173:94: error: invalid new-expression of abstract class type ‘mmdeploy::TRTBicubicInterpolate’
173 | TRTBicubicInterpolate plugin = new TRTBicubicInterpolate(name, scale_factor, align_corners);
| ^
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:175:10: error: cannot convert ‘mmdeploy::TRTBicubicInterpolate
’ to ‘nvinfer1::IPluginV2*’ in return
175 | return plugin;
| ^~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.hpp:9,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp: At global scope:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:83:22: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
83 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:179:68: note: in expansion of macro ‘TRT_NOEXCEPT’
179 | const char name, const void serialData, size_t serialLength) TRT_NOEXCEPT {
| ^~~~~~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.hpp:9,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp: In member function ‘virtual nvinfer1::IPluginV2
mmdeploy::TRTBicubicInterpolateCreator::deserializePlugin(const char
, const void*, size_t)’:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/bicubic_interpolate/trt_bicubic_interpolate.cpp:180:73: error: invalid new-expression of abstract class type ‘mmdeploy::TRTBicubicInterpolate’
180 | auto plugin = new TRTBicubicInterpolate(name, serialData, serialLength);
| ^
make[2]: *** [csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/build.make:258: csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/deform_conv/trt_deform_conv.cpp.o] エラー 1
make[2]: *** [csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/build.make:104: csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/bicubic_interpolate/trt_bicubic_interpolate.cpp.o] エラー 1
make[1]: *** [CMakeFiles/Makefile2:130: csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/all] エラー 2
make: *** [Makefile:136: all] エラー 2

Is this method correct? Is it really that difficult to use TensorRT?
How can I use TensorRT for Mask R-CNN inference using MMDetection? Do I have to manually rewrite all the IPluginV3 code files?

The current version of TensorRT is 10.3.0, the same as when JetPack6.1 was installed (mmdet_env) andolab@andolab:~/mmdet_env$ jetson_release
Software part of jetson-stats 4.3.0 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson Orin NX Engineering Reference Developer Kit - Jetpack 6.1 [L4T 36.4.0]
NV Power Mode[1]: 10W
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:

  • P-Number: p3767-0000
  • Module: NVIDIA Jetson Orin NX (16GB ram)
    Platform:
  • Distribution: Ubuntu 22.04 Jammy Jellyfish
  • Release: 5.15.148-tegra
    jtop:
  • Version: 4.3.0 - Service: Active Libraries: -CUDA: 12.6.68 - cuDNN: 9.3.0.75 - TensorRT: 10.3.0.30 -VPI: 3.2.4 - Vulkan: 1.3.204 - OpenCV: 4.12.0-dev - with CUDA: YES

Do I need to upgrade TensorRT?

Hi,

Usually, we recommended user to convert the pth model with the PyTorch API directly:

You can find an example below:
https://pytorch.org/docs/stable/onnx.html

Then you should be able to convert the ONNX model into TensorRT simply with :

/usr/src/tensorrt/bin/trtexec --onnx=[file]

Below is a related sample which might help:

Thanks.

Hello.
I have shown you the error code before,
but when I try to convert to TensorRT with this code, I get an error saying the plugin cannot be found.
I don’t know how to solve this error.

/usr/src/tensorrt/bin/trtexec --onnx=/home/andolab/mmdet_env/model.onnx

[01/31/2025-12:43:32] [E] [TRT] ModelImporter.cpp:951: — End node —
[01/31/2025-12:43:32] [E] [TRT] ModelImporter.cpp:953: ERROR: onnxOpCheckers.cpp:780 In function checkFallbackPluginImporter:
[6] creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[01/31/2025-12:43:32] [E] [TRT] ModelImporter.cpp:948: While parsing node number 1064 [MMCVMultiLevelRoiAlign → “/mask_roi_extractor/MMCVMultiLevelRoiAlign_output_0”]:
[01/31/2025-12:43:32] [E] [TRT] ModelImporter.cpp:950: — Begin node —
input: “/Reshape_61_output_0”
input: “/neck/fpn_convs.0/conv/Conv_output_0”
input: “/neck/fpn_convs.1/conv/Conv_output_0”
input: “/neck/fpn_convs.2/conv/Conv_output_0”
input: “/neck/fpn_convs.3/conv/Conv_output_0”
output: “/mask_roi_extractor/MMCVMultiLevelRoiAlign_output_0”
name: “/mask_roi_extractor/MMCVMultiLevelRoiAlign”
op_type: “MMCVMultiLevelRoiAlign”
attribute {
name: “aligned”
i: 1
type: INT
}
attribute {
name: “featmap_strides”
floats: 4
floats: 8
floats: 16
floats: 32
type: FLOATS
}
attribute {
name: “finest_scale”
i: 56
type: INT
}
attribute {
name: “output_height”
i: 14
type: INT
}
attribute {
name: “output_width”
i: 14
type: INT
}
attribute {
name: “pool_mode”
i: 1
type: INT
}
attribute {
name: “roi_scale_factor”
f: 1
type: FLOAT
}
attribute {
name: “sampling_ratio”
i: 0
type: INT
}
domain: “mmdeploy”

[01/31/2025-12:43:32] [E] [TRT] ModelImporter.cpp:951: — End node —
[01/31/2025-12:43:32] [E] [TRT] ModelImporter.cpp:953: ERROR: onnxOpCheckers.cpp:780 In function checkFallbackPluginImporter:
[6] creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[01/31/2025-12:43:32] [E] Failed to parse onnx file
[01/31/2025-12:43:32] [I] Finished parsing network model. Parse time: 0.422798
[01/31/2025-12:43:32] [E] Parsing model failed
[01/31/2025-12:43:32] [E] Failed to create engine from model or file.
[01/31/2025-12:43:32] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --onnx=/home/andolab/mmdet_env/model.onnx

I use MMDetection for training and inference, and MMDeploy to convert the Pytorch training model to ONNX format. However, when I use trtexec, I get an error that the plugin cannot be found.

(mmdet_env) andolab@andolab:~/mmdet_env$ python3 /home/andolab/mmdet_env/TensorRT/samples/python/detectron2/build_engine.py --onnx /home/andolab/mmdet_env/model.onnx --engine /home/andolab/mmdet_env/engine.trt --precision fp16
Could not import Detectron 2 modules. Maybe you did not install Detectron 2
Please install Detectron 2, check detectron2/INSTALL.md at main · facebookresearch/detectron2 · GitHub

Hi,

Have you tried to use the PyTorch default ONNX exporter to save the ONNX file?
If not, please give it a try.

It looks like your custom ONNX converter (MMDetection?) has its model definition.

op_type: “MMCVMultiLevelRoiAlign”

The corresponding inference implementation might exist in MMDeploy but will be a problem when using other inference engines like TensorRT.
If your model can be converted to ONNX with a standard PyTorch converter, the layer should be replaced with other mathematical equivalent layers, which, with high possibility, can work with TensorRT.

Thanks.

Hello.

I tried it, but it was not successful.

import torch
import torchvision

学習済み Mask R-CNN をロード

model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False, num_classes=7) # 背景+6クラス
checkpoint = torch.load(“epoch_13.pth”, map_location=“cpu”)
#model.load_state_dict(checkpoint,strict=True)
model.load_state_dict(checkpoint,strict=False)
model.eval()

静的な入力サイズを設定

dummy_input = torch.randn(1, 3, 945, 740, device=“cpu”)

ONNX エクスポート

torch.onnx.export(
model,
dummy_input,
“mask_rcnn.onnx”,
opset_version=11, # Jetson は 11~13 を推奨
input_names=[“input”],
output_names=[“boxes”, “labels”, “scores”, “masks”],
dynamic_axes=None # これで静的形状にする
)

print(“✅ ONNX モデルを静的形状でエクスポートしました!”)

First, when I run this code with model.load_state_dict(checkpoint,strict=True),

Unexpected key(s) in state_dict: “meta”, “state_dict”, “message_hub”, “optimizer”, “param_schedulers”.

This error appears, and when I run it with model.load_state_dict(checkpoint,strict=False),

the export to ONNX format is successful, but

[02/03/2025-13:12:33] [I] === Device Information ===

[02/03/2025-13:12:33] [I] Available Devices:

[02/03/2025-13:12:33] [I] Device 0: “Orin” UUID: GPU-efa7eec7-56ef-5dc0-9ee6-8cbd3607653f [02/03/2025-13:12:33] [I] Selected Device: Orin [02/03/2025-13:12:33] [I] Selected Device ID: 0 [02/03/2025-13:12:33] [I] Selected Device UUID: GPU-efa7eec7-56ef-5dc0-9ee6-8cbd3607653f [02/03/2025-13:12:33] [I] Compute Capability: 8.7 [02/03/2025-13:12:33] [I] SMS: 4 [02/03/2025-13:12:33] [I] Device Global Memory: 15655 MiB [02/03/2025-13:12:33] [I] Shared Memory per SM: 164 KiB [02/03/2025-13:12:33] [I] Memory Bus Width: 256 bits (ECC disabled) [02/03/2025-13:12:33] [I] Application Compute Clock Rate: 0.918 GHz [02/03/2025-13:12:33] [I] Application Memory Clock Rate: 0.612 GHz [02/03/2025-13:12:33] [I] [02/03/2025-13:12:33] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at. [02/03/2025-13:12:33] [I] [02/03/2025-13:12:33] [I] TensorRT version: 10.3.0 [02/03/2025-13:12:33] [I] Loading standard plugins [02/03/2025-13:12:34] [I] [TRT] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 31, GPU 6115 (MiB) [02/03/2025-13:12:36] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +928, GPU +736, now: CPU 1002, GPU 6896 (MiB) [02/03/2025-13:12:36] [I] Start parsing network model. [02/03/2025-13:12:37] [I] [TRT] ---------------------------------------------------------------- [02/03/2025-13:12:37] [I] [TRT] Input filename: mask_rcnn.onnx [02/03/2025-13:12:37] [I] [TRT] ONNX IR version: 0.0.6 [02/03/2025-13:12:37] [I] [TRT] Opset version: 11 [02/03/2025-13:12:37] [I] [TRT] Producer name: pytorch [02/03/2025-13:12:37] [I] [TRT] Producer version: 2.3.0 [02/03/2025-13:12:37] [I] [TRT] Domain: [02/03/2025-13:12:37] [I] [TRT] Model version: 0 [02/03/2025-13:12:37] [I] [TRT] Doc string: [02/03/2025-13:12:37] [I] [TRT] ---------------------------------------------------------------- [02/03/2025-13:12:37] [E] Error[4]: ITensor::getDimensions: Error Code 4: Shape Error (reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,28] to [0,-1].) [02/03/2025-13:12:37] [E] [TRT] ModelImporter.cpp:948: While parsing node number 1657 [Reshape → “/roi_heads/Reshape_output_0”]: [02/03/2025-13:12:37] [E] [TRT] ModelImporter.cpp:950: — Begin node — input: “/roi_heads/box_predictor/bbox_pred/Gemm_output_0” input: “/roi_heads/Concat_output_0” output: “/roi_heads/Reshape_output_0” name: “/roi_heads/Reshape” op_type: “Reshape” [02/03/2025-13:12:37] [E] [TRT] ModelImporter.cpp:951: — End node — [02/03/2025-13:12:37] [E] [TRT] ModelImporter.cpp:953: ERROR: ModelImporter.cpp:195 In function parseNode: [6] Invalid Node - /roi_heads/Reshape ITensor::getDimensions: Error Code 4: Shape Error (reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,28] to [0,-1].) [02/03/2025-13:12:37] [E] Failed to parse onnx file [02/03/2025-13:12:37] [I] Finished parsing network model. Parse time: 0.3444 [02/03/2025-13:12:37] [E] Parsing model failed [02/03/2025-13:12:37] [E] Failed to create engine from model or file. [02/03/2025-13:12:37] [E] Engine set up failed &&&& FAILED TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --onnx=mask_rcnn.onnx --saveEngine=mask_rcnn.trt --fp16

First of all, my training model epoch_13.pth is a Mask R-CNN with a Resnet101 backbone.
I hope I’m mistaken, but it seems that Torchvision does not provide a Mask R-CNN with a Resnet101 backbone.
Also, since I trained it with MMDetection, I feel that the information in the Config file is not reflected in this method.

How can I export the Mask R-CNN with a Resnet101 backbone trained with MMDetection to ONNX format and use it in the TensorRT engine?

from mmdet.apis import init_detector, inference_detector
import torch
import mmcv
import cv2
import time
import numpy as np
#from mmdet.apis import show_result_pyplot
#from mmdet.utils import get_device
from mmdet.utils import register_all_modules
from mmengine.visualization import Visualizer
from mmcv.transforms import Compose
import random

Configファイルと重みファイルのパス

config_file = ‘/home/andolab/mmdet_env/configs/mask_rcnn/my_config.py’

checkpoint_file = ‘/home/andolab/mmdet_env/epoch_13.pth’

モデルを初期化

device = ‘cuda:0’ if torch.cuda.is_available() else ‘cpu’
model = init_detector(config_file, checkpoint_file, device=device)

The above code loads epoch_13.pth and performs Mask R-CNN inference.
The following information is written in this config file.

base = ‘./mask-rcnn_r50_fpn_1x_coco.py’

model = dict(

backbone=dict(

depth=101,

init_cfg=dict(type=‘Pretrained’,

checkpoint=‘torchvision://resnet101’)),

roi_head=dict(

bbox_head=dict(num_classes=6),

mask_head=dict(num_classes=6)))

Define class names (including background classes)

metainfo = {

‘classes’ : (‘T’, ‘L’, ‘V’, ‘C’, ‘N’, ‘Ca’,)

}

When I asked chatGPT, they told me that “the key mismatch occurs because the model trained with MMDetection is trying to load weights into torchvision’s maskrcnn_resnet50_fpn.” Is it possible to convert Mask R-CNN (Resnet101) trained with MMDetection to ONNX?

Hi,

It looks like some specific layers/types are added from MMDetection.
If you need to use it for training, you might also need to use it for inferencing.

Have you tried to use the standard PyTorch to train the MaskRCNN model.
For example:

https://pytorch.org/vision/main/models/mask_rcnn.html

Thanks.

import torch
import torchvision

✅ 1. Mask R-CNN (ResNet101) モデルをロード

num_classes = 7 # 背景 + 6 クラス
model = torchvision.models.detection.maskrcnn_resnet50_fpn(
pretrained=False, num_classes=num_classes, pretrained_backbone=False
)

✅ 2. 学習済みの重みをロード

checkpoint = torch.load(“epoch_13.pth”, map_location=“cpu”)
model.load_state_dict(checkpoint)
model.eval() # 推論モードに設定

✅ 3. ダミー入力を作成(固定サイズ)

dummy_input = torch.randn(1, 3, 945, 740, device=“cpu”)

✅ 4. ONNX にエクスポート(torch.onnx を使用)

torch.onnx.export(
model,
dummy_input,
“mask_rcnn_resnet101.onnx”,
opset_version=11, # Jetson の TensorRT に最適な opset_version
input_names=[“input”],
output_names=[“boxes”, “labels”, “scores”, “masks”],
dynamic_axes=None # 動的形状を無効化し、固定サイズにする
)

print(“✅ Mask R-CNN (ResNet101) を ONNX 形式に変換完了!”)

As I mentioned before, it seems that torchvision only supports mask_rcnn_resnet50_fpn, and does not support my mask_rcnn_resnet101_fpn.I get an error that the keys don’t match.

ing_mean", “backbone.body.layer4.2.bn3.running_var”, “backbone.fpn.inner_blocks.0.0.weight”, “backbone.fpn.inner_blocks.0.0.bias”, “backbone.fpn.inner_blocks.1.0.weight”, “backbone.fpn.inner_blocks.1.0.bias”, “backbone.fpn.inner_blocks.2.0.weight”, “backbone.fpn.inner_blocks.2.0.bias”, “backbone.fpn.inner_blocks.3.0.weight”, “backbone.fpn.inner_blocks.3.0.bias”, “backbone.fpn.layer_blocks.0.0.weight”, “backbone.fpn.layer_blocks.0.0.bias”, “backbone.fpn.layer_blocks.1.0.weight”, “backbone.fpn.layer_blocks.1.0.bias”, “backbone.fpn.layer_blocks.2.0.weight”, “backbone.fpn.layer_blocks.2.0.bias”, “backbone.fpn.layer_blocks.3.0.weight”, “backbone.fpn.layer_blocks.3.0.bias”, “rpn.head.conv.0.0.weight”, “rpn.head.conv.0.0.bias”, “rpn.head.cls_logits.weight”, “rpn.head.cls_logits.bias”, “rpn.head.bbox_pred.weight”, “rpn.head.bbox_pred.bias”, “roi_heads.box_head.fc6.weight”, “roi_heads.box_head.fc6.bias”, “roi_heads.box_head.fc7.weight”, “roi_heads.box_head.fc7.bias”, “roi_heads.box_predictor.cls_score.weight”, “roi_heads.box_predictor.cls_score.bias”, “roi_heads.box_predictor.bbox_pred.weight”, “roi_heads.box_predictor.bbox_pred.bias”, “roi_heads.mask_head.0.0.weight”, “roi_heads.mask_head.0.0.bias”, “roi_heads.mask_head.1.0.weight”, “roi_heads.mask_head.1.0.bias”, “roi_heads.mask_head.2.0.weight”, “roi_heads.mask_head.2.0.bias”, “roi_heads.mask_head.3.0.weight”, “roi_heads.mask_head.3.0.bias”, “roi_heads.mask_predictor.conv5_mask.weight”, “roi_heads.mask_predictor.conv5_mask.bias”, “roi_heads.mask_predictor.mask_fcn_logits.weight”, “roi_heads.mask_predictor.mask_fcn_logits.bias”.
Unexpected key(s) in state_dict: “meta”, “state_dict”, “message_hub”, “optimizer”, “param_schedulers”.

Also, it seems that my model uses a library called MMCV from MMDetection to preprocess the input images.
So I used MMdeploy to convert it to ONNX, but I’m having trouble converting it to TensorRT.

(mmdet_env) andolab@andolab:~/mmdet_env/mmdeploy/build2$ cmake -DMMDEPLOY_TARGET_BACKENDS=trt -DTENSORRT_DIR=/usr/lib/aarch64-linux-gnu -DTENSORRT_VERSION_COMPATIBLE=ON …

When I run Make with this Cmake setting, I get the following error.
The MMdeploy plugin for TensorRT fails to build.
IPluginV2 is deprecated, and I’ve been told to change to IPluginV3, but I don’t have the knowledge to do that.
What should I do?
I asked on MMdeploy’s Git, but no one answered me.

In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:63:76: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
63 | size_t serialLength) TRT_NOEXCEPT override;
| ^~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: ‘IPluginV2DynamicExt’ is deprecated [-Wdeprecated-declarations]
11 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:28:61: note: in expansion of macro ‘TRT_NOEXCEPT’
28 | nvinfer1::IPluginV2DynamicExt *GridPriorsTRT::clone() const TRT_NOEXCEPT {
| ^~~~~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntime.h:403:22: note: declared here
403 | class TRT_DEPRECATED IPluginV2DynamicExt : public nvinfer1::IPluginV2Ext
| ^~~~~~~~~~~~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
11 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:122:66: note: in expansion of macro ‘TRT_NOEXCEPT’
122 | const char *name, const nvinfer1::PluginFieldCollection *fc) TRT_NOEXCEPT {
| ^~~~~~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
11 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:148:83: note: in expansion of macro ‘TRT_NOEXCEPT’
148 | size_t serialLength) TRT_NOEXCEPT {
| ^~~~~~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_priors/trt_grid_priors.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
[ 13%] Building CXX object csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler.cpp.o
cd /home/andolab/mmdet_env/mmdeploy/build2/csrc/mmdeploy/backend_ops/tensorrt && /usr/bin/c++ -DTHRUST_IGNORE_DEPRECATED_CPP_DIALECT=1 -I/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/…/common -I/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common -I/usr/local/cuda-12.6/include -I/include -DMMDEPLOY_USE_CUDA=1 -O3 -fPIC -std=gnu++14 -MD -MT csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler.cpp.o -MF CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler.cpp.o.d -o CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler.cpp.o -c /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp
[ 15%] Building CUDA object csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler_kernel.cu.o
cd /home/andolab/mmdet_env/mmdeploy/build2/csrc/mmdeploy/backend_ops/tensorrt && /usr/local/cuda-12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/usr/bin/c++ -DTHRUST_IGNORE_DEPRECATED_CPP_DIALECT=1 -I/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/…/common -I/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common -I/usr/local/cuda-12.6/include -I/include -Xcompiler=-fPIC,-Wall,-fvisibility=hidden -Xcompiler=-fno-gnu-unique -gencode arch=compute_52,code=sm_52 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -O3 -Xcompiler=-fPIC -std=c++14 -MD -MT csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler_kernel.cu.o -MF CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler_kernel.cu.o.d -x cu -c /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler_kernel.cu -o CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler_kernel.cu.o
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:25:61: warning: ‘IPluginV2DynamicExt’ is deprecated [-Wdeprecated-declarations]
25 | er1::IPluginV2DynamicExt *clone() const TRT_NOEXCEPT override;
| ^~~~~~~~

In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntime.h:403:22: note: declared here
403 | class TRT_DEPRECATED IPluginV2DynamicExt : public nvinfer1::IPluginV2Ext
| ^~~~~~~~~~~~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:78:20: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
78 | TRT_NOEXCEPT override;
| ^~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:81:76: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
81 | size_t serialLength) TRT_NOEXCEPT override;
| ^~~~~~~~

In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: ‘IPluginV2DynamicExt’ is deprecated [-Wdeprecated-declarations]
11 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:29:62: note: in expansion of macro ‘TRT_NOEXCEPT’
29 | IPluginV2DynamicExt *TRTGridSampler::clone() const TRT_NOEXCEPT {
| ^~~~~~~~~~~~

In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntime.h:403:22: note: declared here
403 | class TRT_DEPRECATED IPluginV2DynamicExt : public nvinfer1::IPluginV2Ext
| ^~~~~~~~~~~~~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp: In member function ‘virtual int mmdeploy::TRTGridSampler::enqueue(const nvinfer1::PluginTensorDesc*, const nvinfer1::PluginTensorDesc*, const void* const*, void* const*, void*, cudaStream_t)’:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:113:26: error: cannot convert ‘int64_t*’ {aka ‘long int*’} to ‘int*’
113 | &(output_dims.d[0]), &(input_dims.d[0]), &(grid_dims.d[0]),
| ^~~~~~~~~~~~~~~~~~~
| |
| int64_t* {aka long int*}
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:8:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler_kernel.hpp:10:65: note: initializing argument 4 of ‘void grid_sample(T*, const T*, const T*, int*, int*, int*, int, GridSamplerInterpolation, GridSamplerPadding, bool, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]’
10 | ple(T *output, const T *input, const T *grid, int *output_dims, int *input_dims,
| ~^~~~~~~

In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp: At global scope:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
11 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:162:66: note: in expansion of macro ‘TRT_NOEXCEPT’
162 | *name, const nvinfer1::PluginFieldCollection *fc) TRT_NOEXCEPT {
| ^~~~~~~~~~~~

In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
In file included from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
11 | define TRT_NOEXCEPT noexcept
| ^~~~~~~~
/home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:193:84: note: in expansion of macro ‘TRT_NOEXCEPT’
193 | size_t serialLength) TRT_NOEXCEPT {
| ^~~~~~~~~~~~

In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:34,
from /usr/include/aarch64-linux-gnu/NvInferImpl.h:22,
from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:27,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:4,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.hpp:10,
from /home/andolab/mmdet_env/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/grid_sampler/trt_grid_sampler.cpp:2:
/usr/include/aarch64-linux-gnu/NvInferRuntimePlugin.h:127:22: note: declared here
127 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
make[2]: *** [csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/build.make:356: csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/grid_sampler/trt_grid_sampler.cpp.o] エラー 1
make[2]: *** 未完了のジョブを待っています…
make[2]: ディレクトリ ‘/home/andolab/mmdet_env/mmdeploy/build2’ から出ます
make[1]: *** [CMakeFiles/Makefile2:130: csrc/mmdeploy/backend_ops/tensorrt/CMakeFiles/mmdeploy_tensorrt_ops_obj.dir/all] エラー 2
make[1]: ディレクトリ ‘/home/andolab/mmdet_env/mmdeploy/build2’ から出ます
make: *** [Makefile:136: all] エラー 2

Hi,

Just double-check the MMDeploy.

It looks like they do have the TensorRT support.
Although you might need to run it with MMDeploy if some custom layers are inserted.

Does the 1.2 fps contain an end-to-end pipeline or just inference?
Could you try to benchmark the inference performance and share it with us?

https://github.com/open-mmlab/mmdeploy/blob/main/demo/python/object_detection.py

    for i in range(loop):
        detector = Detector(
            model_path=args.model_path, device_name=args.device_name, device_id=0)
            bboxes, labels, masks = detector(img)

Thanks.