TensorRT partially corrupt output for batch size >1

Description

I am running an EAST text detection model converted to TensorRT (from PyTorch via ONNX).
This model has 2 outputs: “score” and “geo”.
For a batch size of 1 the output is as expected. For a batch size >1, the output of the first image is correct, for all other images in the batch “score” is correct, but “geo” is partially corrupted and partially correct.
This happens even when re-ordering the input images: it is always the images after the first in the batch that are partially corrupted.

Environment

TensorRT Version: 7.2.2.3 (installed using the Python wheel file installation method). (also tested on 7.0.0.11 using tar file installation)
GPU Type: RTX2080-Ti
Nvidia Driver Version: 460.39
CUDA Version: 11.1 (10.2 for the 7.0.0.11 test)
CUDNN Version: 8.1.0.27
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.8 (3.7 for 7.0.0.11 test)
PyTorch Version (if applicable): 1.6.0

Relevant Files

The onnx file of the model and 4 test images (from the public KvasirV2 dataset) that exhibit the behavior is available here: tensorrt_test - Google Drive
I have also created a small example code snippet that demonstrates the behavior in the same folder.

Steps To Reproduce

  • Run the python script from drive.
  • View the output in the terminal. This should give all “true”, but only for the “geo” output, a few values are “false”
    image

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
    No errors are thrown, the output is simply partially incorrect.

Hi @sander.klomp,

Please allow us sometime to work on this issue.

Thank you.

Hi,

Have you guys managed to reproduce the issue or is it likely to be an issue on my side?

Thanks for your time.