Incorrect inference results after converting from ONNX to TRT with trtexec

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
#1

Description

A model converted from ONNX to TensorRT does not produce the correct inference results. A dynamically-shaped TF model for pose estimation (MoveNet multipose) was converted to ONNX format using tf2onnx converter. I verified that the ONNX model gives the same output as the TF model. I converted the ONNX model to TRT format using trtexec. The generated TRT model does not give the same output as the ONNX model: the model output is the correct shape, but the values are incorrect.

Can you please provide guidance to try and get the TRT model working correctly?

Environment

TensorRT Version: 8.5.7.1
GPU Type: RTX 3060 (Laptop)
Nvidia Driver Version: Studio Driver 522.30
CUDA Version: 11.7 & 11.8
CUDNN Version: 8.6.0
Operating System + Version: Windows 10 (19044.2251)
Python Version (if applicable): 3.9.6
Baremetal or Container (if container which image + tag): Bare metal

Relevant Files

The files necessary to reproduce this are in this Google Drive

Steps To Reproduce

  1. Download the ONNX model to the working directory.
  2. Use trtexec as follows: ‘C:\Program Files\NVIDIA\TensorRT\v8.5.1.7\bin\trtexec.exe’ --onnx=model.onnx --saveEngine=model.trt --shapes=input:1x192x256x3
  3. Run the test script with both models on a test image whose shape is close to 192x256.
  4. The two models produce different results.
#2

Hi,

We are unable to successfully run the script you’ve shared in the command line. Facing QT related errors.
However, we could reproduce the accuracy difference between TensorRT and ONNX-Runtime. Please allow us some time to work on this.

[V] Loaded Module: tensorrt | Version: 8.5.0.12 | Path: ['/usr/local/lib/python3.8/dist-packages/tensorrt']

[I] Accuracy Comparison | trt-runner-N0-11/15/22-05:35:13 vs. onnxrt-runner-N0-11/15/22-05:35:13

[I]
[E]         FAILED | Output: 'output_0' | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[E]     FAILED | Mismatched outputs: ['output_0']
[E] Accuracy Summary | trt-runner-N0-11/15/22-05:35:13 vs. onnxrt-runner-N0-11/15/22-05:35:13 | Passed: 0/1 iterations | Pass Rate: 0.0%
[E] FAILED | Runtime: 68.875s | Command: /usr/local/bin/polygraphy run model.onnx --trt --onnxrt --workspace=200G --verbose

Thank you.

#6

Thank you for getting back to me.

I received an email (doesn’t seem to be shown on this thread for some reason) suggesting I check out the following documentation:

and recommending that I “write new plugins or refactor existing ones to target the IPluginV2DynamicExt or IPluginV2IOExt interfaces instead.”.

I’m not sure how this helps me - could you please elaborate a little? I’m relatively new to deep learning in general, let alone TensorRT, so any extra guidance would be greatly appreciated!

#7

Hi,

Did you manage to make any progress with my issue?

#8

Make sure you are loading the correctly trained model.

Another thing you can try to investigate is try your code on a simpler model you know it works.