i have trained a model with pytorch and export it to onnx format model, this onnx model predicts correct value. but when i convert it to tensorrt engine, it gives wrong inference.
i use polygraphy to check the difference of the two. my command is :
polygraphy run detector_corrector.onnx --trt --onnxrt
here is the output:
test_diff.output (236.2 KB)
i use the image of tensorrt :
nvcr.io/nvidia/tensorrt 22.09-py3 dcf7a448b6aa
here is my model:
TensorRT Version: 22.214.171.124
GPU Type: NVIDIA GeForce RTX 3090
Nvidia Driver Version: 470.129.06
CUDA Version: cuda_11.8.r11.8
CUDNN Version: 8
Operating System + Version: Ubuntu 20.04.5 LTS
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.12.1
Baremetal or Container (if container which image + tag):
Hi @liwenjudetiankong ,
Apologies for delayed response.
Please allow us sometime to check on this.
Thank you for your patience.
Could you please elaborate more on the above highlighted.
Are you looking for 100% numerical accuracy in comparison with ONNX-Runtime results?
Or do you have a benchmarking metric for inference accuracy?
Our engineers think the polygraphy issue is just a tolerance issue. With “–atol 1e-4 --rtol 1e-4”, the polygraphy check could get passed, while the default value is “–atol 1e-5 --rtol 1e-5”.
please reference this post, Tensorrt8.5 inference different with origin onnx model
i have upload test code, onnx and trt runtime just give totally diffrent result.
Could you also please give more info on requested.
Based on our debugging, we do not think this is a TRT issue.
It is simply a floating-point arithmetic error; the user should include a postprocessing script in the code and run it on a real dataset, comparing the output sentence rather than the numeric output.