Tensorrt8.5 inference different with origin onnx model


i have trained a model with pytorch and export it to onnx format model, this onnx model predicts correct value. but when i convert it to tensorrt engine, it gives wrong inference.
i use polygraphy to check the difference of the two. my command is :
polygraphy run detector_corrector.onnx --trt --onnxrt

here is the output:
test_diff.output (236.2 KB)

i use the image of tensorrt :
nvcr.io/nvidia/tensorrt 22.09-py3 dcf7a448b6aa

here is my model:


TensorRT Version:
GPU Type: NVIDIA GeForce RTX 3090
Nvidia Driver Version: 470.129.06
CUDA Version: cuda_11.8.r11.8
CUDNN Version: 8
Operating System + Version: Ubuntu 20.04.5 LTS
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.12.1
Baremetal or Container (if container which image + tag):

Hi @liwenjudetiankong ,
Apologies for delayed response.
Please allow us sometime to check on this.
Thank you for your patience.

Hi @liwenjudetiankong,

Could you please elaborate more on the above highlighted.
Are you looking for 100% numerical accuracy in comparison with ONNX-Runtime results?
Or do you have a benchmarking metric for inference accuracy?

Our engineers think the polygraphy issue is just a tolerance issue. With “–atol 1e-4 --rtol 1e-4”, the polygraphy check could get passed, while the default value is “–atol 1e-5 --rtol 1e-5”.

Thank you.

please reference this post, Tensorrt8.5 inference different with origin onnx model
i have upload test code, onnx and trt runtime just give totally diffrent result.

Hi @liwenjudetiankong,

Could you also please give more info on requested.

Thank you.

Based on our debugging, we do not think this is a TRT issue.
It is simply a floating-point arithmetic error; the user should include a postprocessing script in the code and run it on a real dataset, comparing the output sentence rather than the numeric output.

Thank you.