How to deal with conversion error from torch to tensorrt

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi, community. I converted my pytorch model with custom layer from pytroch to tensorrt through torch2trt (GitHub - NVIDIA-AI-IOT/torch2trt: An easy to use PyTorch to TensorRT converter).
So I write custom plugin for tensorrt and custom converter for torch2trt.

To keep it simple I took lenet and convert it to tensorrt and after that measured the error like this:

# create some regular pytorch model...
model = LeNet.eval().cuda()

# create example data
x = torch.ones((1, 1, 32, 32)).cuda()

# convert to TensorRT feeding sample data as input
model_trt = torch2trt(model, [x])
y = model(x)
y_trt = model_trt(x)

# check the output against PyTorch
print(torch.max(torch.abs(y - y_trt)))

And the error is very small like 4e-7/
After that I tried to convert lenet with my custom layers and the error is like 0.018

But when I load weights and measure the error it becomes even higher.

model_path = 'models/lenet5_mnist.pt'
lenet5_model.load_state_dict(torch.load(model_path))
lenet5_model.eval().cuda()

# create example data
x = torch.ones((1, 1, 32, 32)).cuda()

# convert to TensorRT feeding sample data as input
model_trt = torch2trt(lenet5_model, [x]) 


y = lenet5_model(x)
print(torch.max(torch.abs(y - y_trt)))

And the error is around 2.5:

tensor(2.5567, device='cuda:0', grad_fn=<MaxBackward1>)

I’ve tried inference on several images and make sure that there is a great drop in accuracy.

I’m trying to debug this and wanted to print architecture.
for pytorch model I can simple write:

print(lenet5_model)

but when I try on wrapped tensorrt model:

print(model_trt)

it outputs the only:

TRTModule()

Tried to check model weights, for torch is ok:

print(lenet5_model.state_dict()['_body.0.weight'])

But tensorrt outputs smth unreadable:

print(model_trt.state_dict())

So my question is how to check model graph, weights etc in tensorrt?
Or may be some tips how to debug my convertation?

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Thanks for sharing. Although I’m converting model directly from pytorch to tensorrt using torch2trt, not ONNX

5

Hi,

We recommend you to please post your concern on Issues · NVIDIA-AI-IOT/torch2trt · GitHub to get better help.

Thank you.