TensorRT 10.4 inference gives incorrect results (correct when using v8.6)

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Description

TensorRT 10.4 gives incorrect inference output for ViT ONNX model . I am testing using trtexec on NVIDIA Tesla V-100.

I get correct results when using TensorRT 8.6 with the same ONNX file.

( I fist run into this issue when using onnxruntime-gpu=1.19.2 with tensorrt=10.4.0 . I tested with trtexec to see if the issue is due to onnxruntime-gpu or TensorRT )

Environment

TensorRT Version: 10.4.0
GPU Type: Tesla V-100
Nvidia Driver Version: 535.183.01
CUDA Version: 12.2
CUDNN Version: 9
Operating System + Version: Ubuntu 22.04
Python Version (if applicable): 3.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 2.4.1
Baremetal or Container (if container which image + tag):

Relevant Files

I have included the ONNX model, test input , and corresponding PyTorch output (matches trtexec output when using TensorRT 8.6 but not for TensorRT 10.4 )

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Running it using the following bash script

export BIN_DIR=$TENSORRT_ROOT/bin
export ONNX_MODEL_PATH="./vit_c1_h1024_w128.onnx"

$BIN_DIR/trtexec \
--onnx=$ONNX_MODEL_PATH \
--verbose \
--dumpOutput \
--iterations=1 \
--loadInputs='input' \
--noTF32

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered