Inference time increases in for loop

jnaranjo · February 2, 2023, 8:34am

Description

Hi, I am using this repo (https://github.com/wang-xinyu/tensorrtx/blob/master/) to convert a yolov5 network to .engine model.

I have an issue regarding inference time. If I follow this script: wang-xinyu/tensorrtx/blob/master/yolov5/yolov5_det_trt.py

I can see a slow but gradual increase in inference time. Is the solution in a for loop optimal?
Is it recommended to do another procedure to predict a large number of images?

I am very beginner when it comes to optimising models with respect to inference time.

Thanks in advance

Environment

TensorRT Version: TensorRT: 8.4.0.11
GPU Type: Jetson Orin
Nvidia Driver Version:
CUDA Version: 8.7
CUDNN Version: 8.3.2.49
Operating System + Version: 20.04.5 LTS (Focal Fossa)"
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/l4t-pytorch:r34.1.1-pth1.12-py3

Relevant Files

github.com

wang-xinyu/tensorrtx/blob/master/yolov5/yolov5_det_trt.py

"""
An example that uses TensorRT's Python api to make inferences.
"""
import ctypes
import os
import shutil
import random
import sys
import threading
import time
import cv2
import numpy as np
import pycuda.autoinit
import pycuda.driver as cuda
import tensorrt as trt

CONF_THRESH = 0.5
IOU_THRESHOLD = 0.4
LEN_ALL_RESULT = 38001
LEN_ONE_RESULT = 38

This file has been truncated. show original

Steps To Reproduce

Explained in tensorrtx/yolov5 at master · wang-xinyu/tensorrtx · GitHub

spolisetty · February 6, 2023, 5:19am

Hi,

Please try the Nvidia official sample and let us know if you still have this problem.

Also we recommend you to please use the latest TensorRT version 8.5.3.

Thank you.

jnaranjo · February 6, 2023, 7:06am

Hi,
thanks for the reply. I am using yolov5 not yolov3. Is there a tutorial to convert this model (in .pt format) to TensorRT in an official way?
Thanks in advance

Topic		Replies	Views
Inference time increases in for loop cuDNN tensorrt , jetson-inference , yolo , python	2	843	January 31, 2023
Inference time of tensorrt 6.3 is slower than tensorrt 6.0 TensorRT tensorrt , driveos	7	915	October 12, 2021
The problem of time - consuming jump appears in TensorRT 7.0 accelerated yolov5s model reasoning TensorRT	5	582	August 20, 2020
Inference time is linear respective to batch size while using TENSORRT MODEL TensorRT tensorrt , yolo	8	2829	May 5, 2021
TensorRT inference time increase TensorRT cuda	5	458	April 5, 2021
Inference time increases rapidly when set a high resolution input image TensorRT tensorrt , cuda , ubuntu	1	784	September 13, 2023
The inference speed of yolov5 tensorrt has little difference between int8 and fp16 TensorRT tensorrt , cuda	1	1481	September 8, 2022
Inference time using TF-TRT is the same as Native Tensorflow for Object Detection Models TensorRT tensorrt , tf-trt	4	1008	March 31, 2022
Building a engine takes too long TensorRT	13	3219	December 8, 2022
Inconsistent TensorRT Inference Time on Jetson Xavier NX TensorRT	5	26	March 4, 2025

Inference time increases in for loop

Description

Environment

Relevant Files

Steps To Reproduce

Related topics