ONNX Model and Tensorrt Engine gives different output

Description

I have exported a PyTorch model to ONNX and the output matches, which means the ONNX model seems to be working as expected. However, after generating Tensorrt Engine from this ONNX file the outputs are different.

Environment

TensorRT Version: 8.0.1.6
GPU Type: Tesla V100-PCIE-16GB-LS
Nvidia Driver Version: 470.129.06
CUDA Version: 10.2
CUDNN Version: CUDNN_MAJOR 8
Operating System + Version: Ubuntu 18.04.6 LTS
Python Version (if applicable): python3.6
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

https://drive.google.com/drive/folders/1-vsBozRxrbQ0WDDcMlBMpjOEXEC_ryTc?usp=sharing

Steps To Reproduce

python3 compare_pytorch_tensorrt.py

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

I have already shared the ONNX and TensorRT models in their respective files. Please download both the models and scripts using the link provided.

Have there been any updates?

I found the problem may be caused by the operation of TopK. I set the k as 614. It seems a large K will result in error as showed in history posts.