Iplugin tensorrt engine error for ds5.0

@ashingtsai
I have tried Jetson AGX + TensorRT 7.1.0, there is no error as you mentioned reported.
It seems there are shape layers in the generated ONNX.
shape layers are not expected to exist in ONNX generated from https://github.com/Tianxiaomo/pytorch-YOLOv4.
Are there any updates in your local YOLOv4 model?
Do you mind providing your ONNX file so that we can check what are the differences?

@ersheng

you can download my .oonx from below linker.
It is original Yolov4.cfg and yolov4.weight from [https://raw.githubusercontent.com/AlexeyAB]

1 Like

The ONNX module of pytorch 1.5 seems to behave differently from earlier pytorch versions while dealing with constant parameters for expand operations.
Try to generate onnx file with pytorch 1.4 or pytorch 1.3.

Please see compatible pytorch version in TensorRT 7 release note: https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-7.html

Pytorch && ONNX are evolving quickly and we are trying best to catch up.
Tell me if tensorRT reports error again.

@ersheng

yes,I changed to use pytorch V1.4 then can get yolov4.engine successeful from onnx.
then,How to inference yolov4.engine ? in ./trtexec ?
and how to inference it with DLA?
python example is better.C++ also is fine.

@ashingtsai @jiejing_ma

We have not finalize inference methods for YOLOv4 yet.
But you can try to debug and run the following piece of python script alongside with TianXiaomo’s repository: https://github.com/Tianxiaomo/pytorch-YOLOv4 :

trt_demo.py  <engine_file> <image_file> <input_H> <input_W>

However, this TensorRT demo still depends on TianXiaomo’s python functions to do the post-processing that would be a bottle net for inference.

Boosted post-processing for YOLOv4 is still under development.

# trt_demo.py
import sys
import os
import argparse
import numpy as np
import cv2
from PIL import Image
import common
import tensorrt as trt

from tool.utils import *

TRT_LOGGER = trt.Logger()

def main(engine_path, image_path, image_size):
    with get_engine(engine_path) as engine, engine.create_execution_context() as context:
        buffers = common.allocate_buffers(engine)
        image_src = cv2.imread(image_path)

        detect(engine, context, buffers, image_src, image_size)


def get_engine(engine_path):
    # If a serialized engine exists, use it instead of building an engine.
    print("Reading engine from file {}".format(engine_path))
    with open(engine_path, "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
        return runtime.deserialize_cuda_engine(f.read())



def detect(engine, context, buffers, image_src, image_size):
    IN_IMAGE_H, IN_IMAGE_W = image_size

    # Input
    resized = cv2.resize(image_src, (IN_IMAGE_W, IN_IMAGE_H), interpolation=cv2.INTER_LINEAR)
    img_in = cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)
    img_in = np.transpose(img_in, (2, 0, 1)).astype(np.float32)
    img_in = np.expand_dims(img_in, axis=0)
    img_in /= 255.0
    img_in = np.ascontiguousarray(img_in)
    print("Shape of the network input: ", img_in.shape)
    # print(img_in)

    inputs, outputs, bindings, stream = buffers
    print('Length of inputs: ', len(inputs))
    inputs[0].host = img_in

    trt_outputs = common.do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)

    print('Shape of outputs: ')
    print(trt_outputs[0].shape)
    print(trt_outputs[1].shape)
    print(trt_outputs[2].shape)

    trt_outputs[0] = trt_outputs[0].reshape(-1, 255, IN_IMAGE_H // 8, IN_IMAGE_W // 8)
    trt_outputs[1] = trt_outputs[1].reshape(-1, 255, IN_IMAGE_H // 16, IN_IMAGE_W // 16)
    trt_outputs[2] = trt_outputs[2].reshape(-1, 255, IN_IMAGE_H // 32, IN_IMAGE_W // 32)

    print('Shapes supposed to be: ')
    print(trt_outputs[0].shape)
    print(trt_outputs[1].shape)
    print(trt_outputs[2].shape)

    # print(outputs[2])
    num_classes = 80

    boxes = post_processing(img_in, 0.4, num_classes, 0.5, trt_outputs)

    if num_classes == 20:
        namesfile = 'data/voc.names'
    elif num_classes == 80:
        namesfile = 'data/coco.names'
    else:
        namesfile = 'data/names'

    class_names = load_class_names(namesfile)
    plot_boxes_cv2(image_src, boxes, savename='predictions_trt.jpg', class_names=class_names)



if __name__ == '__main__':
    engine_path = sys.argv[1]
    image_path = sys.argv[2]
    
    if len(sys.argv) < 4:
        image_size = (416, 416)
    elif len(sys.argv) < 5:
        image_size = (int(sys.argv[3]), int(sys.argv[3]))
    else:
        image_size = (int(sys.argv[3]), int(sys.argv[4]))
    
    main(engine_path, image_path, image_size)

@ashingtsai @jiejing_ma

Since https://github.com/Tianxiaomo/pytorch-YOLOv4 is now evolving quickly with my contributions, my previous post has been phased out already.
Please pull from the latest https://github.com/Tianxiaomo/pytorch-YOLOv4 and follow guidelines in README.
If you have problems with conversions and inferences, do not hesitate to inform me.

1 Like

@ersheng

I run below command but got error.
python3 demo_trt.py yolov4.engine dog.jpg

Reading engine from file yolov4.engine
Shape of the network input: (1, 3, 416, 416)
Length of inputs: 1
Len of outputs: 9
Shapes supposed to be:
(8112,)
(32448,)
(648960,)
(2028,)
(8112,)
(162240,)
(507,)
(40560,)
(2028,)
Traceback (most recent call last):
File “demo_trt.py”, line 228, in
main(engine_path, image_path, image_size)
File “demo_trt.py”, line 117, in main
detect(engine, context, buffers, image_src, image_size)
File “demo_trt.py”, line 184, in detect
trt_outputs[1].reshape(-1, 3 * h1 * w1, 80),
ValueError: cannot reshape array of size 32448 into shape (8112,80)

@ashingtsai
Order of outputs is often randomized.
I have tried to solve this problem and you can pull the latest code.

@ersheng

then it seems got inf value in box.

python3 demo_trt.py yolov4.engine dog.jpg
Reading engine from file yolov4.engine
Shape of the network input: (1, 3, 416, 416)
Length of inputs: 1
Len of outputs: 9
/home/ashing/pytorch-YOLOv4/tool/utils.py:39: RuntimeWarning: invalid value encountered in add
My = max(box1[1] + box1[3] / 2.0, box2[1] + box2[3] / 2.0)
Traceback (most recent call last):
File “demo_trt.py”, line 235, in
main(engine_path, image_path, image_size)
File “demo_trt.py”, line 117, in main
detect(engine, context, buffers, image_src, image_size)
File “demo_trt.py”, line 220, in detect
plot_boxes_cv2(image_src, boxes, savename=‘predictions_trt.jpg’, class_names=class_names)
File “/home/ashing/pytorch-YOLOv4/tool/utils.py”, line 490, in plot_boxes_cv2
y1 = int((box[1] - box[3] / 2.0) * height)
OverflowError: cannot convert float infinity to integer

@ashingtsai
exp() in yolo layer may cause infinite values.
Send me your input image (dog.jpg) and let me reproduce this fault.
Or maybe you can try this by yourself.

In the second last line of function yolo_forward:
from

boxes = torch.cat((xmin, ymin, xmax, ymax), dim=2)

to

boxes = torch.cat((xmin, ymin, xmax, ymax), dim=2).clamp(-10.0, 10.0)

@ersheng

I try to change to
boxes = torch.cat((xmin, ymin, xmax, ymax), dim=2).clamp(-10.0, 10.0)
but no any change ,still appear the same error.
and the dog.jpg is from pytorch-YOLOv4/data/dog.jpg
my pytorch version is 1.4

@ashingtsai
Which platform is this float infinity error from?
Jetson NX?
I have tried on Tesla T4 with pytorch 1.4 and TensorRT 7.0, there is no problem.

@ersheng

yes,I run it on Jetson NX.

@ersheng

thank you for your reply.
seems I can get the right result now.
but I can not sure what I really do.
Maybe relative I rebuild trtexec with pytorch 1.4 onnx 1.6 or something.
should be rebuild trtexec by
make TARGET=aarch64
however,I still can not see any result on detection picture.
(predictions_trt.jpg)
I will try to debug it ,too.

python3 demo_trt.py yolov4.trt dog.jpg
Reading engine from file yolov4.trt
Shape of the network input: (1, 3, 416, 416)
Length of inputs: 1
Len of outputs: 9
truck: 0.980496
dog: 0.999999
dog: 0.999999
bicycle: 1.000000
bicycle: 1.000000
truck: 0.946777
bicycle: 1.000000
truck: 0.951660
truck: 0.947754
dog: 1.000000
dog: 1.000000
bicycle: 0.999999
bicycle: 1.000000
bicycle: 1.000000
bicycle: 1.000000
bicycle: 1.000000
bicycle: 1.000000
save plot results to predictions_trt.jpg

@ashingtsai

You can pull the latest README and source from https://github.com/Tianxiaomo/pytorch-YOLOv4.

YoloV4 TensorRT demo can run on X86 smoothly.
Send me information if you still have problems running YoloV4 on Jetson platforms.

@ersheng

better ,but incorrect.

Google Photos

hi @mchi
Can you share it with me please,
I have been trying it for a while now
thanks

Hi @pinktree3,
Eric has shared to you in previous comment - Iplugin tensorrt engine error for ds5.0

1 Like

The batchsize is fixed when exporting darknet weights to onnx model, what if I want to inference different numbers of videos (dynamic batch size) using this engine file with deepstream?

exporting onnx with this command:

python demo_darknet2onnx.py cfg/yolov4-hat.cfg yolov4-hat_7000.weights 233.png 1

export to trt engine:

/usr/src/tensorrt/bin/trtexec --onnx=yolov4_1_3_416_416_static.onnx --explicitBatch --minShapes=input:1x3x416x416 --optShapes=input:32x3x416x416 --maxShapes=input:32x3x416x416 --workspace=2048 --saveEngine=yolov4-hat.engine --fp16

When I deploy with deepstream, error info:

0:00:20.552723060 12988 0x5608854d5e40 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1642> [UID = 1]: Backend has maxBatchSize 1 whereas 24 has been requested
0:00:20.552736568 12988 0x5608854d5e40 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1813> [UID = 1]: deserialized backend context :/opt/nvidia/deepstream/deepstream-5.0/sources/Yolov4-hat/yolov4-hat.engine failed to match config params, trying rebuild
0:00:20.557183440 12988 0x5608854d5e40 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:934 failed to build network since there is no model file matched.
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:872 failed to build network.
0:00:20.557493110 12988 0x5608854d5e40 ERROR                nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1735> [UID = 1]: build engine file failed
0:00:20.557510437 12988 0x5608854d5e40 ERROR                nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1821> [UID = 1]: build backend context failed
0:00:20.557520575 12988 0x5608854d5e40 ERROR                nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1148> [UID = 1]: generate backend failed, check config file settings
0:00:20.557626494 12988 0x5608854d5e40 WARN                 nvinfer gstnvinfer.cpp:809:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:20.557635838 12988 0x5608854d5e40 WARN                 nvinfer gstnvinfer.cpp:809:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-5.0/sources/Yolov4-hat/config_infer_primary_yoloV4.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
** ERROR: <main:655>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance

The batch size in my deepstream app config is 24.

Hi CoderJustin,

Please help to open a new topic for your issue. Thanks