Inference time increases in for loop

jnaranjo · January 27, 2023, 8:02am

Hi, I am using this repo (https://github.com/wang-xinyu/tensorrtx/blob/master/) to convert a yolov5 network to .engine model.

I have an issue regarding inference time. If I follow this script:

github.com

wang-xinyu/tensorrtx/blob/master/yolov5/yolov5_det_trt.py

"""
An example that uses TensorRT's Python api to make inferences.
"""
import ctypes
import os
import shutil
import random
import sys
import threading
import time
import cv2
import numpy as np
import pycuda.autoinit
import pycuda.driver as cuda
import tensorrt as trt

CONF_THRESH = 0.5
IOU_THRESHOLD = 0.4
LEN_ALL_RESULT = 38001
LEN_ONE_RESULT = 38

This file has been truncated. show original

I can see a slow but gradual increase in inference time. Is the solution in a for loop optimal?
Is it recommended to do another procedure to predict a number of images?

I am very beginner when it comes to optimising models with respect to inference time.

Thanks in advance

AakankshaS · January 31, 2023, 6:53am

Hi @jnaranjo ,
TRT Forum should be able to assist better here.

Thanks

jnaranjo · January 31, 2023, 8:35am

Perfect! Thanks for the reply!