Failed converting ONNX model to TensorRT model

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi Team,

Looking for some help please.

I have an ONNX model (pytorch). I want to convert the model from ONNX to TensorRT, manually and programmatically.
I have written some Python code that uses the TensorRT builder API to do the conversion, and i have tested the code on two different machines/environment:

  1. Nvidia Tesla K80 (AWS P2.xlarge instance)
  2. Nvidia Tesla V100 (AWS P3.xlarge instance)

I can successfully convert the model to TRT on Nvidia V100, programmatically.
But when i run the same conversion code, it fails on my K80 machine. See error log below.

python dependencies:

pip freeze | ag nvidia                                                         
nvidia-cublas-cu11==2022.4.8
nvidia-cublas-cu117==11.10.1.25
nvidia-cuda-runtime-cu11==2022.4.25
nvidia-cuda-runtime-cu117==11.7.60
nvidia-cudnn-cu11==2022.5.19
nvidia-cudnn-cu116==8.4.0.27
nvidia-pyindex==1.0.9
nvidia-tensorrt==8.4.0.6

Log:

[06/07/2022-14:22:18] [TRT] [I] [MemUsageChange] Init CUDA: CPU +152, GPU +0, now: CPU 302, GPU 116 (MiB)
[06/07/2022-14:22:20] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 322 MiB, GPU 116 MiB
[06/07/2022-14:22:20] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 323 MiB, GPU 116 MiB
[06/07/2022-14:22:20] [TRT] [I] ----------------------------------------------------------------
[06/07/2022-14:22:20] [TRT] [I] Input filename:   /tmp/pytest-of-ubuntu/pytest-2/test_mmdetection_model_inferen0/model/saved_model.onnx
[06/07/2022-14:22:20] [TRT] [I] ONNX IR version:  0.0.7
[06/07/2022-14:22:20] [TRT] [I] Opset version:    11
[06/07/2022-14:22:20] [TRT] [I] Producer name:    pytorch
[06/07/2022-14:22:20] [TRT] [I] Producer version: 1.10
[06/07/2022-14:22:20] [TRT] [I] Domain:           
[06/07/2022-14:22:20] [TRT] [I] Model version:    0
[06/07/2022-14:22:20] [TRT] [I] Doc string:       
[06/07/2022-14:22:20] [TRT] [I] ----------------------------------------------------------------
[06/07/2022-14:22:20] [TRT] [W] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[06/07/2022-14:22:20] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:20] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:20] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:20] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:21] [TRT] [I] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-14:22:21] [TRT] [I] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:21] [TRT] [W] builtin_op_importers.cpp:4777: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-14:22:21] [TRT] [I] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-14:22:23] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[06/07/2022-14:22:23] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[06/07/2022-14:22:24] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[06/07/2022-14:22:25] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[06/07/2022-14:22:26] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[06/07/2022-14:22:28] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[06/07/2022-14:22:28] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace: 
[06/07/2022-14:22:28] [TRT] [I] Successfully created plugin: TRTBatchedNMS
[06/07/2022-14:22:31] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.8.0 but loaded cuBLAS/cuBLAS LT 111.0.1
[06/07/2022-14:22:31] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +191, GPU +74, now: CPU 686, GPU 190 (MiB)
[06/07/2022-14:22:31] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +88, GPU +34, now: CPU 774, GPU 224 (MiB)
[06/07/2022-14:22:31] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[06/07/2022-14:22:33] [TRT] [W] GPU error during getBestTactic: Conv_6 + Relu_7 : an illegal memory access was encountered
[06/07/2022-14:22:33] [TRT] [E] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[06/07/2022-14:22:33] [TRT] [E] 1: [virtualMemoryBuffer.cpp::~StdVirtualMemoryBufferImpl::121] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[06/07/2022-14:22:33] [TRT] [E] 10: [optimizer.cpp::computeCosts::2033] Error Code 10: Internal Error (Could not find any implementation for node Conv_6 + Relu_7.)
[06/07/2022-14:22:33] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::619] Error Code 2: Internal Error (Assertion engine != nullptr failed. )

Then i also tried using the trtexec to convert it on K80, but it also failed. See error log below.
I also tried increase the workspace size to 11GB, but also got the same error.

trtexec 's verbose log (partial log):

[06/07/2022-14:44:42] [V] [TRT] Tactic: 10944511 Time: 0.063244
[06/07/2022-14:44:42] [V] [TRT] Fastest Tactic: 2097151 Time: 0.024228
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: Conv_457 (CudnnConvolution)
[06/07/2022-14:44:42] [V] [TRT] Tactic: 0 Time: 0.064372
[06/07/2022-14:44:42] [V] [TRT] Tactic: 1 skipped. Scratch requested: 1024, available: 0
[06/07/2022-14:44:42] [V] [TRT] Tactic: 2 skipped. Scratch requested: 262144, available: 0
[06/07/2022-14:44:42] [V] [TRT] Tactic: 4 skipped. Scratch requested: 11225088, available: 0
[06/07/2022-14:44:42] [V] [TRT] Tactic: 5 skipped. Scratch requested: 1223168, available: 0
[06/07/2022-14:44:42] [V] [TRT] Fastest Tactic: 0 Time: 0.064372
[06/07/2022-14:44:42] [V] [TRT] Setting workspace to 1024enables more tactics for profiling
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: Conv_457 (CublasConvolution)
[06/07/2022-14:44:42] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: Conv_457 (CaskConvolution)
[06/07/2022-14:44:42] [V] [TRT] CaskConvolution has no valid tactics for this config, skipping
[06/07/2022-14:44:42] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: FusedConvActConvolution Tactic: 2097151
[06/07/2022-14:44:42] [V] [TRT] *************** Autotuning format combination: Float(65536,1,4096,256) -> Float(4608,1,288,18) ***************
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: Conv_457 (CudnnConvolution)
[06/07/2022-14:44:42] [V] [TRT] CudnnConvolution has no valid tactics for this config, skipping
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: Conv_457 (CublasConvolution)
[06/07/2022-14:44:42] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: Conv_457 (CaskConvolution)
[06/07/2022-14:44:42] [V] [TRT] CaskConvolution has no valid tactics for this config, skipping
[06/07/2022-14:44:42] [V] [TRT] =============== Computing costs for
[06/07/2022-14:44:42] [V] [TRT] *************** Autotuning format combination: Float(1016,1) -> Float(1000,1), Int32(1000,1) ***************
[06/07/2022-14:44:42] [V] [TRT] --------------- Timing Runner: TopK_1603 (TopK)
[06/07/2022-14:44:42] [V] [TRT] Tactic: 2 skipped. Scratch requested: 512000, available: 0
[06/07/2022-14:44:42] [V] [TRT] Fastest Tactic: -3360065831133338131 Time: inf
[06/07/2022-14:44:42] [V] [TRT] Deleting timing cache: 481 entries, 1232 hits
[06/07/2022-14:44:42] [E] Error[10]: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node TopK_1603.)
[06/07/2022-14:44:42] [E] Error[2]: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
[06/07/2022-14:44:42] [E] Engine could not be created from network
[06/07/2022-14:44:42] [E] Building engine failed
[06/07/2022-14:44:42] [E] Failed to create engine from model.
[06/07/2022-14:44:42] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8400] # ./trtexec --onnx=/home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx --minShapes=input:1x3x244x244 --optShapes=input:1x3x244x244 --maxShapes=input:1x3x244x244 --plugins=/home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so --workspace=4096 --verbose

trtexec 's non verbose log (full log):

./trtexec --onnx=/home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx --minShapes=input:1x3x244x244 --optShapes=input:1x3x244x244 --maxShapes=input:1x3x244x244 --plugins=/home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so
&&&& RUNNING TensorRT.trtexec [TensorRT v8400] # ./trtexec --onnx=/home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx --minShapes=input:1x3x244x244 --optShapes=input:1x3x244x244 --maxShapes=input:1x3x244x244 --plugins=/home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so
[06/07/2022-18:33:09] [I] === Model Options ===
[06/07/2022-18:33:09] [I] Format: ONNX
[06/07/2022-18:33:09] [I] Model: /home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx
[06/07/2022-18:33:09] [I] Output:
[06/07/2022-18:33:09] [I] === Build Options ===
[06/07/2022-18:33:09] [I] Max batch: explicit batch
[06/07/2022-18:33:09] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[06/07/2022-18:33:09] [I] minTiming: 1
[06/07/2022-18:33:09] [I] avgTiming: 8
[06/07/2022-18:33:09] [I] Precision: FP32
[06/07/2022-18:33:09] [I] LayerPrecisions:
[06/07/2022-18:33:09] [I] Calibration:
[06/07/2022-18:33:09] [I] Refit: Disabled
[06/07/2022-18:33:09] [I] Sparsity: Disabled
[06/07/2022-18:33:09] [I] Safe mode: Disabled
[06/07/2022-18:33:09] [I] DirectIO mode: Disabled
[06/07/2022-18:33:09] [I] Restricted mode: Disabled
[06/07/2022-18:33:09] [I] Save engine:
[06/07/2022-18:33:09] [I] Load engine:
[06/07/2022-18:33:09] [I] Profiling verbosity: 0
[06/07/2022-18:33:09] [I] Tactic sources: Using default tactic sources
[06/07/2022-18:33:09] [I] timingCacheMode: local
[06/07/2022-18:33:09] [I] timingCacheFile:
[06/07/2022-18:33:09] [I] Input(s)s format: fp32:CHW
[06/07/2022-18:33:09] [I] Output(s)s format: fp32:CHW
[06/07/2022-18:33:09] [I] Input build shape: input=1x3x244x244+1x3x244x244+1x3x244x244
[06/07/2022-18:33:09] [I] Input calibration shapes: model
[06/07/2022-18:33:09] [I] === System Options ===
[06/07/2022-18:33:09] [I] Device: 0
[06/07/2022-18:33:09] [I] DLACore:
[06/07/2022-18:33:09] [I] Plugins: /home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so
[06/07/2022-18:33:09] [I] === Inference Options ===
[06/07/2022-18:33:09] [I] Batch: Explicit
[06/07/2022-18:33:09] [I] Input inference shape: input=1x3x244x244
[06/07/2022-18:33:09] [I] Iterations: 10
[06/07/2022-18:33:09] [I] Duration: 3s (+ 200ms warm up)
[06/07/2022-18:33:09] [I] Sleep time: 0ms
[06/07/2022-18:33:09] [I] Idle time: 0ms
[06/07/2022-18:33:09] [I] Streams: 1
[06/07/2022-18:33:09] [I] ExposeDMA: Disabled
[06/07/2022-18:33:09] [I] Data transfers: Enabled
[06/07/2022-18:33:09] [I] Spin-wait: Disabled
[06/07/2022-18:33:09] [I] Multithreading: Disabled
[06/07/2022-18:33:09] [I] CUDA Graph: Disabled
[06/07/2022-18:33:09] [I] Separate profiling: Disabled
[06/07/2022-18:33:09] [I] Time Deserialize: Disabled
[06/07/2022-18:33:09] [I] Time Refit: Disabled
[06/07/2022-18:33:09] [I] Skip inference: Disabled
[06/07/2022-18:33:09] [I] Inputs:
[06/07/2022-18:33:09] [I] === Reporting Options ===
[06/07/2022-18:33:09] [I] Verbose: Disabled
[06/07/2022-18:33:09] [I] Averages: 10 inferences
[06/07/2022-18:33:09] [I] Percentile: 99
[06/07/2022-18:33:09] [I] Dump refittable layers:Disabled
[06/07/2022-18:33:09] [I] Dump output: Disabled
[06/07/2022-18:33:09] [I] Profile: Disabled
[06/07/2022-18:33:09] [I] Export timing to JSON file:
[06/07/2022-18:33:09] [I] Export output to JSON file:
[06/07/2022-18:33:09] [I] Export profile to JSON file:
[06/07/2022-18:33:09] [I]
[06/07/2022-18:33:09] [I] === Device Information ===
[06/07/2022-18:33:09] [I] Selected Device: Tesla K80
[06/07/2022-18:33:09] [I] Compute Capability: 3.7
[06/07/2022-18:33:09] [I] SMs: 13
[06/07/2022-18:33:09] [I] Compute Clock Rate: 0.8235 GHz
[06/07/2022-18:33:09] [I] Device Global Memory: 11441 MiB
[06/07/2022-18:33:09] [I] Shared Memory per SM: 112 KiB
[06/07/2022-18:33:09] [I] Memory Bus Width: 384 bits (ECC enabled)
[06/07/2022-18:33:09] [I] Memory Clock Rate: 2.505 GHz
[06/07/2022-18:33:09] [I]
[06/07/2022-18:33:09] [I] TensorRT version: 8.4.0
[06/07/2022-18:33:09] [I] Loading supplied plugin library: /home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so
[06/07/2022-18:33:09] [I] [TRT] [MemUsageChange] Init CUDA: CPU +156, GPU +0, now: CPU 167, GPU 115 (MiB)
[06/07/2022-18:33:10] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 167 MiB, GPU 115 MiB
[06/07/2022-18:33:10] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 230 MiB, GPU 115 MiB
[06/07/2022-18:33:10] [I] Start parsing network model
[06/07/2022-18:33:10] [I] [TRT] ----------------------------------------------------------------
[06/07/2022-18:33:10] [I] [TRT] Input filename:   /home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx
[06/07/2022-18:33:10] [I] [TRT] ONNX IR version:  0.0.7
[06/07/2022-18:33:10] [I] [TRT] Opset version:    11
[06/07/2022-18:33:10] [I] [TRT] Producer name:    pytorch
[06/07/2022-18:33:10] [I] [TRT] Producer version: 1.10
[06/07/2022-18:33:10] [I] [TRT] Domain:
[06/07/2022-18:33:10] [I] [TRT] Model version:    0
[06/07/2022-18:33:10] [I] [TRT] Doc string:
[06/07/2022-18:33:10] [I] [TRT] ----------------------------------------------------------------
[06/07/2022-18:33:10] [W] [TRT] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[06/07/2022-18:33:10] [W] [TRT] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:10] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:10] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:10] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:10] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:11] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:11] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:11] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:11] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:12] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:12] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:12] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:12] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: TRTInstanceNormalization
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:13] [I] [TRT] No importer registered for op: MMCVDeformConv2d. Attempting to import as plugin.
[06/07/2022-18:33:13] [I] [TRT] Searching for plugin: MMCVDeformConv2d, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:13] [W] [TRT] builtin_op_importers.cpp:4780: Attribute deform_groups not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[06/07/2022-18:33:13] [I] [TRT] Successfully created plugin: MMCVDeformConv2d
[06/07/2022-18:33:17] [W] [TRT] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[06/07/2022-18:33:19] [W] [TRT] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[06/07/2022-18:33:22] [W] [TRT] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[06/07/2022-18:33:24] [W] [TRT] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[06/07/2022-18:33:27] [W] [TRT] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped

[06/07/2022-18:33:31] [I] [TRT] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[06/07/2022-18:33:31] [I] [TRT] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace:
[06/07/2022-18:33:31] [I] [TRT] Successfully created plugin: TRTBatchedNMS
[06/07/2022-18:33:31] [W] [TRT] Output type must be INT32 for shape outputs
[06/07/2022-18:33:31] [W] [TRT] Output type must be INT32 for shape outputs
[06/07/2022-18:33:31] [W] [TRT] Output type must be INT32 for shape outputs
[06/07/2022-18:33:31] [W] [TRT] Output type must be INT32 for shape outputs
[06/07/2022-18:33:31] [W] [TRT] Output type must be INT32 for shape outputs
[06/07/2022-18:33:31] [I] Finish parsing network model
[06/07/2022-18:33:33] [W] [TRT] TensorRT was linked against cuBLAS/cuBLASLt 11.6.5 but loaded cuBLAS/cuBLASLt 11.4.1
[06/07/2022-18:33:33] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +123, GPU +49, now: CPU 525, GPU 164 (MiB)
[06/07/2022-18:33:33] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +89, GPU +34, now: CPU 614, GPU 198 (MiB)
[06/07/2022-18:33:33] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[06/07/2022-18:33:42] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[06/07/2022-18:35:20] [E] Error[10]: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node TopK_1603.)
[06/07/2022-18:35:20] [E] Error[2]: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
[06/07/2022-18:35:20] [E] Engine could not be created from network
[06/07/2022-18:35:20] [E] Building engine failed
[06/07/2022-18:35:20] [E] Failed to create engine from model.
[06/07/2022-18:35:20] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8400] # ./trtexec --onnx=/home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx --minShapes=input:1x3x244x244 --optShapes=input:1x3x244x244 --maxShapes=input:1x3x244x244 --plugins=/home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so

btw, i also opened an issue in here: Failed converting ONNX model to TensorRT model · Issue #2039 · NVIDIA/TensorRT · GitHub

Environment

TensorRT Version : 8.4.0.6
NVIDIA GPU : K80
NVIDIA Driver Version : 470.129.06
CUDA Version : cuda_11.2.r11.2/compiler.29618528_0
CUDNN Version : 8.4.1.50
Operating System : ubuntu 20.04
Python Version (if applicable) : 3.7
Tensorflow Version (if applicable) :
PyTorch Version (if applicable) : 1.10.1+cu113
Baremetal or Container (if so, version) : Baremetal (AWS P2 instance)

Relevant Files

ONNX model file: resnet50_reppoints_reflective_screw_od.onnx - Google Drive
Plugin file:
libmmdeploy_tensorrt_ops.so - Google Drive

Steps To Reproduce

Followed the instructions here to build trtexec , and then run the following command:

./trtexec --onnx=/home/ubuntu/work/work_dir/resnet50_reppoints_reflective_screw_od.onnx --minShapes=input:1x3x244x244 --optShapes=input:1x3x244x244 --maxShapes=input:1x3x244x244 --plugins=/home/ubuntu/work/repos/avi-inference/avi_inference/libs/libmmdeploy_tensorrt_ops.so --workspace=4096 --verbose

See the ONNX model file and the plugin file in the “Relevant Files” section.

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

5

Thanks @NVES.
I did both. No errors found in check_modoel.py.
I can see errors in the execution of trtexec.
The error is included in the post above.
The model and plugin file are also included.

6

Hi,

As suggested in the git issue please try downgrading your CUDA to 10.2.

Thank you.