Description
I have two tensorrt engines one for segmentation and detection. When I run them individually I get an FPS of 440 and 367 respectively on 2080 Ti. But When I ran the same two models in two separate threads as mentioned in the thread (How to use TensorRT by the multi-threading package of python - #7 by Chieh), I get an FPS of 534 and 429 respectively. Which is like a 20% increase.
I was expecting the FPS to drop because now the resource is shared between multiple threads or at the best would remain at the same FPS. Increased FPS is good, but it is breaking my head without finding the reason
Environment
TensorRT Version: 7.1.3.4
GPU Type: 2080 Tu
Nvidia Driver Version: 450.51.06
CUDA Version: 11.0
CUDNN Version:
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/tensorrt:20.08-py3