How to deal with conversion error from torch to tensorrt

Hi, community. I converted my pytorch model with custom layer from pytroch to tensorrt through torch2trt (GitHub - NVIDIA-AI-IOT/torch2trt: An easy to use PyTorch to TensorRT converter).
So I write custom plugin for tensorrt and custom converter for torch2trt.

To keep it simple I took lenet and convert it to tensorrt and after that measured the error like this:

# create some regular pytorch model...
model = LeNet.eval().cuda()

# create example data
x = torch.ones((1, 1, 32, 32)).cuda()

# convert to TensorRT feeding sample data as input
model_trt = torch2trt(model, [x])
y = model(x)
y_trt = model_trt(x)

# check the output against PyTorch
print(torch.max(torch.abs(y - y_trt)))

And the error is very small like 4e-7/
After that I tried to convert lenet with my custom layers and the error is like 0.018

But when I load weights and measure the error it becomes even higher.

model_path = 'models/'

# create example data
x = torch.ones((1, 1, 32, 32)).cuda()

# convert to TensorRT feeding sample data as input
model_trt = torch2trt(lenet5_model, [x]) 

y = lenet5_model(x)
print(torch.max(torch.abs(y - y_trt)))

And the error is around 2.5:

tensor(2.5567, device='cuda:0', grad_fn=<MaxBackward1>)

I’ve tried inference on several images and make sure that there is a great drop in accuracy.

I’m trying to debug this and wanted to print architecture.
for pytorch model I can simple write:


but when I try on wrapped tensorrt model:


it outputs the only:


Tried to check model weights, for torch is ok:


But tensorrt outputs smth unreadable:


So my question is how to check model graph, weights etc in tensorrt?
Or may be some tips how to debug my convertation?

Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
2) Try running your model with trtexec command.
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging

Thanks for sharing. Although I’m converting model directly from pytorch to tensorrt using torch2trt, not ONNX


We recommend you to please post your concern on Issues · NVIDIA-AI-IOT/torch2trt · GitHub to get better help.

Thank you.