Converting ONNX Model(Which has two inputs) to TRT #1020

Hi Nvidia Team,

I have converted the Slow Fast model(trained on our Custom dataset) from Pt to ONNX. I need to further convert the ONNX model to TRT. But the main challenge is that the model has two Inputs(As shown in the image below). I usually convert the ONNX model to TRT using the standard code below:

import tensorrt as trt
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
trt_runtime = trt.Runtime(TRT_LOGGER)
def build_engine(onnx_path, shape = [1,3,112,112]):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(1) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.fp16_mode = True
        builder.max_workspace_size = (256 << 20)
        with open(onnx_path, 'rb') as model:
            parser.parse(model.read())
        network.get_input(0).shape = shape
        engine = builder.build_cuda_engine(network)
        return engine
def save_engine(engine, file_name):
    buf = engine.serialize()
    with open(file_name, 'wb') as f:
        f.write(buf)
def load_engine(trt_runtime, plan_path):
    with open(engine_path, 'rb') as f:
        engine_data = f.read()
    engine = trt_runtime.deserialize_cuda_engine(engine_data)
    return engine
from onnx import ModelProto
 
engine_name = "model.trt"
onnx_path = "model.onnx"
 
model = ModelProto()
with open(onnx_path, "rb") as f:
    model.ParseFromString(f.read())
 
batch_size = 1
 
d0 = model.graph.input[0].type.tensor_type.shape.dim[1].dim_value
d1 = model.graph.input[0].type.tensor_type.shape.dim[2].dim_value
d2 = model.graph.input[0].type.tensor_type.shape.dim[3].dim_value
shape = [batch_size , d0, d1 ,d2]
 
engine = build_engine(onnx_path)
save_engine(engine, engine_name) 

But, as I mentioned above that the Slowfast model has two inputs, I am facing difficulty in converting the ONNX Model(with two inputs) to TRT.

I Kindly request you assist me in changing the above Code for two Inputs for Conversion.

Thanks in advance,
Darshan

Hi, Request you to share the ONNX model and the script so that we can assist you better.

Alongside you can try validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).

Alternatively, you can try running your model with trtexec command.

Thanks!

Hi @NVES,

Thank you very much for your quick reply.
ONNX Model: https://drive.google.com/file/d/1ToSD6gKkquI58iCcCK3-5S5kyeRmR86i/view?usp=sharing

Pt to ONNX Conversion Code:

import io
import numpy as np
import torch.onnx
from defaults import get_cfg
import model_builder




cfg = get_cfg()
model = model_builder.build_model(cfg)
fast_pathway= torch.randn(1,3,32,256,455)
slow_pathway= torch.randn(1, 3,8,256,455)
inputs=[slow_pathway, fast_pathway]
for p in model.parameters():
   p.requires_grad = False
device = "cuda:0"
model = model.to(device)
model.eval()

torch.onnx.export(model,inputs,"slow_fast.onnx", opset_version=12)          

ONNX to TRT Conversion Script:

import tensorrt as trt
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
trt_runtime = trt.Runtime(TRT_LOGGER)
def build_engine(onnx_path, shape = [1,3,112,112]):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(1) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.fp16_mode = True
        builder.max_workspace_size = (256 << 20)
        with open(onnx_path, 'rb') as model:
            parser.parse(model.read())
        network.get_input(0).shape = shape
        engine = builder.build_cuda_engine(network)
        return engine
def save_engine(engine, file_name):
    buf = engine.serialize()
    with open(file_name, 'wb') as f:
        f.write(buf)
def load_engine(trt_runtime, plan_path):
    with open(engine_path, 'rb') as f:
        engine_data = f.read()
    engine = trt_runtime.deserialize_cuda_engine(engine_data)
    return engine
from onnx import ModelProto
 
engine_name = "slow_fast.onnx.trt"
onnx_path = "slow_fast.onnx.onnx"
 
model = ModelProto()
with open(onnx_path, "rb") as f:
    model.ParseFromString(f.read())
 
batch_size = 1
 
d0 = model.graph.input[0].type.tensor_type.shape.dim[1].dim_value
d1 = model.graph.input[0].type.tensor_type.shape.dim[2].dim_value
d2 = model.graph.input[0].type.tensor_type.shape.dim[3].dim_value
shape = [batch_size , d0, d1 ,d2]
 
engine = build_engine(onnx_path)
save_engine(engine, engine_name)

Above I have attached the ONNX Model and Code for Converting from Pt to ONNX as well as from ONNX to TRT.

Thanks Again

Hi @NVES,

I have Validated my Model using the Snippet sent by you:

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model)

And there is no output after running that(Seems like there is no issue with the Model).

Hi @NVES,

Looking forward to your reply.

Thanks

Hi @darshancganji12,

Are you using custom plugin in your code?
Can you please try converting your model using trtexec
trtexec --onnx=your_model.onnx --verbose --explicitBatch --shapes=input1:1x1x1x1,input2:2x2x2x2

Thank you.

Hi @spolisetty,

Thanks for your response.
I have tried to convert the model using the trtexec, the output has been attached below:

&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=/home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx --explicitBatch --shapes=input1:1x1x1x1,input2:2x2x2x2
[01/19/2021-12:06:52] [I] === Model Options ===
[01/19/2021-12:06:52] [I] Format: ONNX
[01/19/2021-12:06:52] [I] Model: /home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx
[01/19/2021-12:06:52] [I] Output:
[01/19/2021-12:06:52] [I] === Build Options ===
[01/19/2021-12:06:52] [I] Max batch: explicit
[01/19/2021-12:06:52] [I] Workspace: 16 MB
[01/19/2021-12:06:52] [I] minTiming: 1
[01/19/2021-12:06:52] [I] avgTiming: 8
[01/19/2021-12:06:52] [I] Precision: FP32
[01/19/2021-12:06:52] [I] Calibration: 
[01/19/2021-12:06:52] [I] Safe mode: Disabled
[01/19/2021-12:06:52] [I] Save engine: 
[01/19/2021-12:06:52] [I] Load engine: 
[01/19/2021-12:06:52] [I] Builder Cache: Enabled
[01/19/2021-12:06:52] [I] NVTX verbosity: 0
[01/19/2021-12:06:52] [I] Inputs format: fp32:CHW
[01/19/2021-12:06:52] [I] Outputs format: fp32:CHW
[01/19/2021-12:06:52] [I] Input build shape: input1=1x1x1x1+1x1x1x1+1x1x1x1
[01/19/2021-12:06:52] [I] Input build shape: input2=2x2x2x2+2x2x2x2+2x2x2x2
[01/19/2021-12:06:52] [I] Input calibration shapes: model
[01/19/2021-12:06:52] [I] === System Options ===
[01/19/2021-12:06:52] [I] Device: 0
[01/19/2021-12:06:52] [I] DLACore: 
[01/19/2021-12:06:52] [I] Plugins:
[01/19/2021-12:06:52] [I] === Inference Options ===
[01/19/2021-12:06:52] [I] Batch: Explicit
[01/19/2021-12:06:52] [I] Input inference shape: input2=2x2x2x2
[01/19/2021-12:06:52] [I] Input inference shape: input1=1x1x1x1
[01/19/2021-12:06:52] [I] Iterations: 10
[01/19/2021-12:06:52] [I] Duration: 3s (+ 200ms warm up)
[01/19/2021-12:06:52] [I] Sleep time: 0ms
[01/19/2021-12:06:52] [I] Streams: 1
[01/19/2021-12:06:52] [I] ExposeDMA: Disabled
[01/19/2021-12:06:52] [I] Spin-wait: Disabled
[01/19/2021-12:06:52] [I] Multithreading: Disabled
[01/19/2021-12:06:52] [I] CUDA Graph: Disabled
[01/19/2021-12:06:52] [I] Skip inference: Disabled
[01/19/2021-12:06:52] [I] Inputs:
[01/19/2021-12:06:52] [I] === Reporting Options ===
[01/19/2021-12:06:52] [I] Verbose: Disabled
[01/19/2021-12:06:52] [I] Averages: 10 inferences
[01/19/2021-12:06:52] [I] Percentile: 99
[01/19/2021-12:06:52] [I] Dump output: Disabled
[01/19/2021-12:06:52] [I] Profile: Disabled
[01/19/2021-12:06:52] [I] Export timing to JSON file: 
[01/19/2021-12:06:52] [I] Export output to JSON file: 
[01/19/2021-12:06:52] [I] Export profile to JSON file: 
[01/19/2021-12:06:52] [I] 
----------------------------------------------------------------
Input filename:   /home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx
ONNX IR version:  0.0.6
Opset version:    12
Producer name:    pytorch
Producer version: 1.7
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[01/19/2021-12:06:53] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/19/2021-12:06:53] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: Einsum. Attempting to import as plugin.
[01/19/2021-12:06:53] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: Einsum, plugin_version: 1, plugin_namespace: 
[01/19/2021-12:06:53] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin Einsum version 1
ERROR: builtin_op_importers.cpp:3661 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[01/19/2021-12:06:53] [E] Failed to parse onnx file
[01/19/2021-12:06:53] [E] Parsing model failed
[01/19/2021-12:06:53] [E] Engine creation failed
[01/19/2021-12:06:53] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # ./trtexec --onnx=/home/lenovo/Slow-Fast/ACTIONRWPMHV2.0_Dec/slow_fast.onnx --explicitBatch --shapes=input1:1x1x1x1,input2:2x2x2x2

Hi @darshancganji12,

Looks like there is some problem with custom plugin. Could you please share error logs you got when tried with python script you mentioned.

Thank you.

Hi @spolisetty,

In ONNX Model, I can find two Ops that are not currently supported in Tensorrt. One is Einsum and another is RoiAlign. Is there any way to overcome these two unsupported Ops? I feel registering a Custom Plugin is quite a difficult task.

Thanks,
Darshan

Hi @darshancganji12,

We cannot overcome unsupported Ops. You need to implement custom Ops.
For your reference,

Thank you.