I am running an EAST text detection model converted to TensorRT (from PyTorch via ONNX).
This model has 2 outputs: “score” and “geo”.
For a batch size of 1 the output is as expected. For a batch size >1, the output of the first image is correct, for all other images in the batch “score” is correct, but “geo” is partially corrupted and partially correct.
This happens even when re-ordering the input images: it is always the images after the first in the batch that are partially corrupted.
TensorRT Version: 220.127.116.11 (installed using the Python wheel file installation method). (also tested on 18.104.22.168 using tar file installation)
GPU Type: RTX2080-Ti
Nvidia Driver Version: 460.39
CUDA Version: 11.1 (10.2 for the 22.214.171.124 test)
CUDNN Version: 126.96.36.199
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.8 (3.7 for 188.8.131.52 test)
PyTorch Version (if applicable): 1.6.0
The onnx file of the model and 4 test images (from the public KvasirV2 dataset) that exhibit the behavior is available here: tensorrt_test - Google Drive
I have also created a small example code snippet that demonstrates the behavior in the same folder.
Steps To Reproduce
- Run the python script from drive.
- View the output in the terminal. This should give all “true”, but only for the “geo” output, a few values are “false”
- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered
No errors are thrown, the output is simply partially incorrect.