Converting ONNX Model(Which has two inputs) to TRT #1020

darshancganji12 · January 18, 2021, 7:33pm

Hi Nvidia Team,

I have converted the Slow Fast model(trained on our Custom dataset) from Pt to ONNX. I need to further convert the ONNX model to TRT. But the main challenge is that the model has two Inputs(As shown in the image below). I usually convert the ONNX model to TRT using the standard code below:

import tensorrt as trt
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
trt_runtime = trt.Runtime(TRT_LOGGER)
def build_engine(onnx_path, shape = [1,3,112,112]):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(1) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.fp16_mode = True
        builder.max_workspace_size = (256 << 20)
        with open(onnx_path, 'rb') as model:
            parser.parse(model.read())
        network.get_input(0).shape = shape
        engine = builder.build_cuda_engine(network)
        return engine
def save_engine(engine, file_name):
    buf = engine.serialize()
    with open(file_name, 'wb') as f:
        f.write(buf)
def load_engine(trt_runtime, plan_path):
    with open(engine_path, 'rb') as f:
        engine_data = f.read()
    engine = trt_runtime.deserialize_cuda_engine(engine_data)
    return engine
from onnx import ModelProto
 
engine_name = "model.trt"
onnx_path = "model.onnx"
 
model = ModelProto()
with open(onnx_path, "rb") as f:
    model.ParseFromString(f.read())
 
batch_size = 1
 
d0 = model.graph.input[0].type.tensor_type.shape.dim[1].dim_value
d1 = model.graph.input[0].type.tensor_type.shape.dim[2].dim_value
d2 = model.graph.input[0].type.tensor_type.shape.dim[3].dim_value
shape = [batch_size , d0, d1 ,d2]
 
engine = build_engine(onnx_path)
save_engine(engine, engine_name)

But, as I mentioned above that the Slowfast model has two inputs, I am facing difficulty in converting the ONNX Model(with two inputs) to TRT.

I Kindly request you assist me in changing the above Code for two Inputs for Conversion.

Thanks in advance,
Darshan

NVES · January 18, 2021, 7:37pm

Hi, Request you to share the ONNX model and the script so that we can assist you better.

Alongside you can try validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).

Alternatively, you can try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

Thanks!

darshancganji12 · January 18, 2021, 7:47pm

Hi @NVES,

Thank you very much for your quick reply.
ONNX Model: slow_fast.onnx - Google Drive

Pt to ONNX Conversion Code:

import io
import numpy as np
import torch.onnx
from defaults import get_cfg
import model_builder




cfg = get_cfg()
model = model_builder.build_model(cfg)
fast_pathway= torch.randn(1,3,32,256,455)
slow_pathway= torch.randn(1, 3,8,256,455)
inputs=[slow_pathway, fast_pathway]
for p in model.parameters():
   p.requires_grad = False
device = "cuda:0"
model = model.to(device)
model.eval()

torch.onnx.export(model,inputs,"slow_fast.onnx", opset_version=12)

ONNX to TRT Conversion Script:

import tensorrt as trt
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
trt_runtime = trt.Runtime(TRT_LOGGER)
def build_engine(onnx_path, shape = [1,3,112,112]):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(1) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.fp16_mode = True
        builder.max_workspace_size = (256 << 20)
        with open(onnx_path, 'rb') as model:
            parser.parse(model.read())
        network.get_input(0).shape = shape
        engine = builder.build_cuda_engine(network)
        return engine
def save_engine(engine, file_name):
    buf = engine.serialize()
    with open(file_name, 'wb') as f:
        f.write(buf)
def load_engine(trt_runtime, plan_path):
    with open(engine_path, 'rb') as f:
        engine_data = f.read()
    engine = trt_runtime.deserialize_cuda_engine(engine_data)
    return engine
from onnx import ModelProto
 
engine_name = "slow_fast.onnx.trt"
onnx_path = "slow_fast.onnx.onnx"
 
model = ModelProto()
with open(onnx_path, "rb") as f:
    model.ParseFromString(f.read())
 
batch_size = 1
 
d0 = model.graph.input[0].type.tensor_type.shape.dim[1].dim_value
d1 = model.graph.input[0].type.tensor_type.shape.dim[2].dim_value
d2 = model.graph.input[0].type.tensor_type.shape.dim[3].dim_value
shape = [batch_size , d0, d1 ,d2]
 
engine = build_engine(onnx_path)
save_engine(engine, engine_name)

Above I have attached the ONNX Model and Code for Converting from Pt to ONNX as well as from ONNX to TRT.

Thanks Again

darshancganji12 · January 18, 2021, 8:00pm

Hi @NVES,

I have Validated my Model using the Snippet sent by you:

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model)

And there is no output after running that(Seems like there is no issue with the Model).

darshancganji12 · January 19, 2021, 5:19am

Hi @NVES,

Looking forward to your reply.

Thanks

spolisetty · January 19, 2021, 6:21am

Hi @darshancganji12,

Are you using custom plugin in your code?
Can you please try converting your model using trtexec
trtexec --onnx=your_model.onnx --verbose --explicitBatch --shapes=input1:1x1x1x1,input2:2x2x2x2

Thank you.

darshancganji12 · January 19, 2021, 6:37am

Hi @spolisetty,

Thanks for your response.
I have tried to convert the model using the trtexec, the output has been attached below:

&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=/home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx --explicitBatch --shapes=input1:1x1x1x1,input2:2x2x2x2
[01/19/2021-12:06:52] [I] === Model Options ===
[01/19/2021-12:06:52] [I] Format: ONNX
[01/19/2021-12:06:52] [I] Model: /home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx
[01/19/2021-12:06:52] [I] Output:
[01/19/2021-12:06:52] [I] === Build Options ===
[01/19/2021-12:06:52] [I] Max batch: explicit
[01/19/2021-12:06:52] [I] Workspace: 16 MB
[01/19/2021-12:06:52] [I] minTiming: 1
[01/19/2021-12:06:52] [I] avgTiming: 8
[01/19/2021-12:06:52] [I] Precision: FP32
[01/19/2021-12:06:52] [I] Calibration: 
[01/19/2021-12:06:52] [I] Safe mode: Disabled
[01/19/2021-12:06:52] [I] Save engine: 
[01/19/2021-12:06:52] [I] Load engine: 
[01/19/2021-12:06:52] [I] Builder Cache: Enabled
[01/19/2021-12:06:52] [I] NVTX verbosity: 0
[01/19/2021-12:06:52] [I] Inputs format: fp32:CHW
[01/19/2021-12:06:52] [I] Outputs format: fp32:CHW
[01/19/2021-12:06:52] [I] Input build shape: input1=1x1x1x1+1x1x1x1+1x1x1x1
[01/19/2021-12:06:52] [I] Input build shape: input2=2x2x2x2+2x2x2x2+2x2x2x2
[01/19/2021-12:06:52] [I] Input calibration shapes: model
[01/19/2021-12:06:52] [I] === System Options ===
[01/19/2021-12:06:52] [I] Device: 0
[01/19/2021-12:06:52] [I] DLACore: 
[01/19/2021-12:06:52] [I] Plugins:
[01/19/2021-12:06:52] [I] === Inference Options ===
[01/19/2021-12:06:52] [I] Batch: Explicit
[01/19/2021-12:06:52] [I] Input inference shape: input2=2x2x2x2
[01/19/2021-12:06:52] [I] Input inference shape: input1=1x1x1x1
[01/19/2021-12:06:52] [I] Iterations: 10
[01/19/2021-12:06:52] [I] Duration: 3s (+ 200ms warm up)
[01/19/2021-12:06:52] [I] Sleep time: 0ms
[01/19/2021-12:06:52] [I] Streams: 1
[01/19/2021-12:06:52] [I] ExposeDMA: Disabled
[01/19/2021-12:06:52] [I] Spin-wait: Disabled
[01/19/2021-12:06:52] [I] Multithreading: Disabled
[01/19/2021-12:06:52] [I] CUDA Graph: Disabled
[01/19/2021-12:06:52] [I] Skip inference: Disabled
[01/19/2021-12:06:52] [I] Inputs:
[01/19/2021-12:06:52] [I] === Reporting Options ===
[01/19/2021-12:06:52] [I] Verbose: Disabled
[01/19/2021-12:06:52] [I] Averages: 10 inferences
[01/19/2021-12:06:52] [I] Percentile: 99
[01/19/2021-12:06:52] [I] Dump output: Disabled
[01/19/2021-12:06:52] [I] Profile: Disabled
[01/19/2021-12:06:52] [I] Export timing to JSON file: 
[01/19/2021-12:06:52] [I] Export output to JSON file: 
[01/19/2021-12:06:52] [I] Export profile to JSON file: 
[01/19/2021-12:06:52] [I] 
----------------------------------------------------------------
Input filename:   /home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx
ONNX IR version:  0.0.6
Opset version:    12
Producer name:    pytorch
Producer version: 1.7
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[01/19/2021-12:06:53] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/19/2021-12:06:53] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: Einsum. Attempting to import as plugin.
[01/19/2021-12:06:53] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: Einsum, plugin_version: 1, plugin_namespace: 
[01/19/2021-12:06:53] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin Einsum version 1
ERROR: builtin_op_importers.cpp:3661 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[01/19/2021-12:06:53] [E] Failed to parse onnx file
[01/19/2021-12:06:53] [E] Parsing model failed
[01/19/2021-12:06:53] [E] Engine creation failed
[01/19/2021-12:06:53] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # ./trtexec --onnx=/home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx --explicitBatch --shapes=input1:1x1x1x1,input2:2x2x2x2

spolisetty · January 19, 2021, 7:14am

Hi @darshancganji12,

Looks like there is some problem with custom plugin. Could you please share error logs you got when tried with python script you mentioned.

Thank you.

darshancganji12 · January 19, 2021, 7:59am

Hi @spolisetty,

In ONNX Model, I can find two Ops that are not currently supported in Tensorrt. One is Einsum and another is RoiAlign. Is there any way to overcome these two unsupported Ops? I feel registering a Custom Plugin is quite a difficult task.

Thanks,
Darshan

spolisetty · January 19, 2021, 9:30am

Hi @darshancganji12,

We cannot overcome unsupported Ops. You need to implement custom Ops.
For your reference,

Thank you.

Topic		Replies	Views
Problem converting TensorFlow 2-> ONNX model to TensorRT Engine (efficientdet_d0) TensorRT	8	1392	November 17, 2022
Cannot convert SSD ONNX model to TensorRT TensorRT tensorrt	15	2354	November 23, 2022
Onnx model to TRT conversion error TensorRT	6	3280	April 15, 2022
[TensorRT] ERROR: Network must have at least one output TensorRT tensorrt	29	2367	September 30, 2021
Issues with torch.nn.ReflectionPad2d(padding) conversion to TRT engine TensorRT tensorrt , pytorch , onnx	21	4178	February 8, 2022
PluginV2Layer must be V2DynamicExt when there are runtime input dimensions TensorRT tensorrt , cuda	6	1016	October 12, 2021
Failed to generate TRT engine from ONNX model ( Error parsing text-format onnx2trt_onnx.ModelProto: 1:1: Invalid control characters encountered in te) Jetson Nano tensorrt	8	3061	October 18, 2021
I do not get any performance improvement after using TensorRT provider for object detection model Jetson Nano tensorrt , onnx	7	1407	July 12, 2022
Use pre-trained object detection TF2 models with TensorRT ONNX TensorRT	9	1923	May 31, 2021
Unsupported ONNX data type: UINT8 (2) TensorRT	24	8918	May 6, 2021

Converting ONNX Model(Which has two inputs) to TRT #1020

Related topics