Building a engine takes too long

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi! I am trying to build yolov7 by compiling it and saving the serialzed trt engine.
However, the process is too slow. Takes 1hour for 256*256 resolution.
Is there anyway to speed up?

Environment

TensorRT Version: 8.2.4.2
GPU Type: RTX3080 12GB
Nvidia Driver Version: 515.48
CUDA Version: 11.4
CUDNN Version: 8.2.4
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.12.0
Baremetal or Container (if container which image + tag):

Relevant Files

here is my onnx file.
https://drive.google.com/file/d/1RnAMCGl-Grft9TDgkn5lcPNvYulIRaPI/view?usp=sharing

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Hi,

We Could not observe similar behavior.

You can try increasing the GPU memory utilization using --workspace option and please refer Developer Guide :: NVIDIA Deep Learning TensorRT Documentation

Also, we recommend you to please try on the latest TensorRT version 8.4 GA Update 1 and if you still face this issue share with us verbose logs and command to try from our end for better debugging.
https://developer.nvidia.com/nvidia-tensorrt-8x-download

Thank you.

5

Here is my log.

https://drive.google.com/file/d/1TXY86Rbk6Fr4CbNkN1ySMrK7mahkYD7t/view?usp=sharing

It starts at 15:51:28 and ends at 16:10:53.
I also set the workspace large enough.
What is the problem?

6

Hi,

Looks like you’re not using trtexec, could you please share with us the issue repro and if possible trtexec --verbose logs.
Also, we recommend you to please try official TensorRT samples for better performance.

Thank you.

7

Yes, I’m experiencing the same thing, running trtexec on yolov7, and it just takes absolutely forever. In fact it’s been running and hasn’t finish after >1hr. This is on a Jetson Xavier dev kit.

8

Hmm, looks like stopping and reloading the trtexec process worked for me, and it rendered an engine 1280x768 in about 20-30 minutes. Still feels a lot slower than older versions, in the previous Jetpack release (4.4), it would take maybe 1-2 minutes to create an engine.

9

Here is trtexec logs.
It’s still slow…

10

could you possibly share trtexec log?
I want to compare it to my log.

11

Hi,

We were trying on a different platform. As you’re using Jetson Xavier, we are moving this post to the Jetson Xavier forum to get better help.

Thank you.

12

No. I am using the x86 platform.
It is also recorded in my trtexec log.

13

I’ve been having much better luck now exporting YOLOv7 models to TensorRT.

Here is a sample log, this one was created using the Python API on an RTX 3090
https://gist.githubusercontent.com/jakepoz/123eb801fa6ae92126403a59ad73eccd/raw/38f21a7f5e129dfee8b8fd668d3a38d9b5ae9259/trtlog.txt

I’m not 100% sure what changed though, after I ran it a few times, the jobs started completing.

14

Hello,I meet the similiar problem.I use the version8.5 of TensorRT to convert the yolov4 onnx to engine model.It still takes 42mins and does not decline.
So I want to know how did you decrease the time of converting the model?

15

I haven’t solved it yet.
What is certain is that the build speed was faster on the gtx 1060.
After replacing the GPU(1060->3080), it became several times slower.