Tutorial: How to run YOLOv7 on Deepstream

vcmike · September 27, 2022, 5:15am

Hi,

To help people run official YOLOv7 models on Deepstream here is some helper code.

You should first export the model to ONNX via this command (taken from the yolov7 README)

python export.py --weights ./yolov7-tiny.pt --grid --end2end --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640

This command will create an ONNX model with an efficientNMS node. To be able to map the model outputs from efficientNMS to the NVIDIA NvDsObjectMeta data structure you need this code which needs to be added/compiled in /opt/nvidia/deepstream/deepstream-6.1/sources/libs/nvdsinfer_customparser/nvdsinfer_custombboxparser.cpp

extern "C"
bool NvDsInferParseCustomEfficientNMS (std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                                   NvDsInferNetworkInfo  const &networkInfo,
                                   NvDsInferParseDetectionParams const &detectionParams,
                                   std::vector<NvDsInferObjectDetectionInfo> &objectList) {
    if(outputLayersInfo.size() != 4)
    {
        std::cerr << "Mismatch in the number of output buffers."
                  << "Expected 4 output buffers, detected in the network :"
                  << outputLayersInfo.size() << std::endl;
        return false;
    }
    const char* log_enable = std::getenv("ENABLE_DEBUG");

    int* p_keep_count = (int *) outputLayersInfo[0].buffer;

    float* p_bboxes = (float *) outputLayersInfo[1].buffer;
    NvDsInferDims inferDims_p_bboxes = outputLayersInfo[1].inferDims;
    int numElements_p_bboxes=inferDims_p_bboxes.numElements;

    float* p_scores = (float *) outputLayersInfo[2].buffer;
    unsigned int* p_classes = (unsigned int *) outputLayersInfo[3].buffer;

    const float threshold = detectionParams.perClassThreshold[0];

    float max_bbox=0;
    for (int i=0; i < numElements_p_bboxes; i++)
    {
        if ( max_bbox < p_bboxes[i] )
            max_bbox=p_bboxes[i];
    }

    if (p_keep_count[0] > 0)
    {
        assert (!(max_bbox < 2.0));
        for (int i = 0; i < p_keep_count[0]; i++) {

            if ( p_scores[i] < threshold) continue;
            assert((unsigned int) p_classes[i] < detectionParams.numClassesConfigured);

            NvDsInferObjectDetectionInfo object;
            object.classId = (int) p_classes[i];
            object.detectionConfidence = p_scores[i];

            object.left=p_bboxes[4*i];
            object.top=p_bboxes[4*i+1];
            object.width=(p_bboxes[4*i+2] - object.left);
            object.height= (p_bboxes[4*i+3] - object.top);

            if(log_enable != NULL && std::stoi(log_enable)) {
                std::cout << "label/conf/ x/y w/h -- "
                << p_classes[i] << " "
                << p_scores[i] << " "
                << object.left << " " << object.top << " " << object.width << " "<< object.height << " "
                << std::endl;
            }

            object.left=CLIP(object.left, 0, networkInfo.width - 1);
            object.top=CLIP(object.top, 0, networkInfo.height - 1);
            object.width=CLIP(object.width, 0, networkInfo.width - 1);
            object.height=CLIP(object.height, 0, networkInfo.height - 1);

            objectList.push_back(object);
        }
    }
    return true;
}

You then just need to make sure your nvinfer stages point to the correct custom library and function like:

parse-bbox-func-name=NvDsInferParseCustomEfficientNMS
custom-lib-path=/opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infercustomparser.so

With that you should be good to go.

yingliu · September 28, 2022, 8:51am

Thanks for your sharing!

xenonscrinium · October 6, 2022, 8:54pm

Hello, could you help me please. I run this example https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-rtsp-in-rtsp-out
with your example.
I performed your instructions. When some object is detected, I get this:

python3: nvdsinfer_custombboxparser.cpp:589: bool NvDsInferParseCustomEfficientNMS(const std::vector&, const NvDsInferNetworkInfo&, const NvDsInferParseDetectionParams&, std::vector&): Assertion `(unsigned int) p_classes[i] < detectionParams.numClassesConfigured’ failed.

img_log

vcmike · October 6, 2022, 9:27pm

Hi @xenonscrinium .

The issue is probably that you have not updated num-detected-classes=91 to be correct for the model you are running. This assertion checks that any detected class from the model is less than this num-detected-classes variable.

zongxp · October 12, 2022, 10:13am

It works, thanks for your sharing.

OforOmar · November 10, 2022, 6:00am

Hello,
I followed the steps you mentioned above but I am not getting any detections, using deepstream_test3.
my config file:-

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=yolov7-tiny.onnx
model-engine-file=yolov7-tiny.onnx_b2_gpu0_fp16.engine
labelfile-path=models/coco.txt
num-detected-classes=80
operate-on-class-ids=0;1;2;9
interval=0
gie-unique-id=1
batch-size=2
network-mode=2
process-mode=1
network-type=2
cluster-mode=2
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomEfficientNMS
custom-lib-path=libnvds_infercustomparser.so

[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
topk=300

vcmike · November 10, 2022, 6:28am

operate-on-class-ids=0;1;2;9 is for secondary inference so might be breaking this?

OforOmar · November 10, 2022, 6:33am

Tried running without this line, nothing still :(

vcmike · November 10, 2022, 6:41am

you could try export ENABLE_DEBUG=1 as it will show if there are actually any detections

philipp.schmidt · November 15, 2022, 9:37am

Anyone willing to share a gst-launch example to try this?

philipp.schmidt · November 15, 2022, 9:57am

Nevermind, works flawlessly!

johann.gerberding · November 16, 2022, 2:53pm

When I run ‘sudo -E make install’ I get the following problems which don’t seem realted to the code posted above (Jeston Xavier NX):

g++ -o libnvds_infercustomparser.so nvdsinfer_custombboxparser.cpp nvdsinfer_customclassifierparser.cpp -Wall -std=c++11 -shared -fPIC -I…/…/includes -I /usr/local/cuda-11.4/include -Wl,–start-group -lnvinfer -lnvparsers -Wl,–end-group
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:19,
from /usr/include/aarch64-linux-gnu/NvInferLegacyDims.h:16,
from /usr/include/aarch64-linux-gnu/NvInfer.h:16,
from /usr/include/aarch64-linux-gnu/NvCaffeParser.h:16,
from …/…/includes/nvdsinfer_custom_impl.h:126,
from nvdsinfer_custombboxparser.cpp:25:
/usr/local/cuda-11.4/include/cuda_runtime_api.h:147:10: fatal error: crt/host_defines.h: No such file or directory
147 | #include “crt/host_defines.h”
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:19,
from /usr/include/aarch64-linux-gnu/NvInferLegacyDims.h:16,
from /usr/include/aarch64-linux-gnu/NvInfer.h:16,
from /usr/include/aarch64-linux-gnu/NvCaffeParser.h:16,
from …/…/includes/nvdsinfer_custom_impl.h:126,
from nvdsinfer_customclassifierparser.cpp:25:
/usr/local/cuda-11.4/include/cuda_runtime_api.h:147:10: fatal error: crt/host_defines.h: No such file or directory
147 | #include “crt/host_defines.h”
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:41: libnvds_infercustomparser.so] Error 1

Any idea?

ahsanmuhammadakram1996 · November 24, 2022, 8:49am

vcmike:

extern "C"
bool NvDsInferParseCustomEfficientNMS (std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                                   NvDsInferNetworkInfo  const &networkInfo,
                                   NvDsInferParseDetectionParams const &detectionParams,
                                   std::vector<NvDsInferObjectDetectionInfo> &objectList) {
    if(outputLayersInfo.size() != 4)
    {
        std::cerr << "Mismatch in the number of output buffers."
                  << "Expected 4 output buffers, detected in the network :"
                  << outputLayersInfo.size() << std::endl;
        return false;
    }
    const char* log_enable = std::getenv("ENABLE_DEBUG");

    int* p_keep_count = (int *) outputLayersInfo[0].buffer;

    float* p_bboxes = (float *) outputLayersInfo[1].buffer;
    NvDsInferDims inferDims_p_bboxes = outputLayersInfo[1].inferDims;
    int numElements_p_bboxes=inferDims_p_bboxes.numElements;

    float* p_scores = (float *) outputLayersInfo[2].buffer;
    unsigned int* p_classes = (unsigned int *) outputLayersInfo[3].buffer;

    const float threshold = detectionParams.perClassThreshold[0];

    float max_bbox=0;
    for (int i=0; i < numElements_p_bboxes; i++)
    {
        if ( max_bbox < p_bboxes[i] )
            max_bbox=p_bboxes[i];
    }

    if (p_keep_count[0] > 0)
    {
        assert (!(max_bbox < 2.0));
        for (int i = 0; i < p_keep_count[0]; i++) {

            if ( p_scores[i] < threshold) continue;
            assert((unsigned int) p_classes[i] < detectionParams.numClassesConfigured);

            NvDsInferObjectDetectionInfo object;
            object.classId = (int) p_classes[i];
            object.detectionConfidence = p_scores[i];

            object.left=p_bboxes[4*i];
            object.top=p_bboxes[4*i+1];
            object.width=(p_bboxes[4*i+2] - object.left);
            object.height= (p_bboxes[4*i+3] - object.top);

            if(log_enable != NULL && std::stoi(log_enable)) {
                std::cout << "label/conf/ x/y w/h -- "
                << p_classes[i] << " "
                << p_scores[i] << " "
                << object.left << " " << object.top << " " << object.width << " "<< object.height << " "
                << std::endl;
            }

            object.left=CLIP(object.left, 0, networkInfo.width - 1);
            object.top=CLIP(object.top, 0, networkInfo.height - 1);
            object.width=CLIP(object.width, 0, networkInfo.width - 1);
            object.height=CLIP(object.height, 0, networkInfo.height - 1);

            objectList.push_back(object);
        }
    }
    return true;
}

I was able to run the model using the provided tutorial on x86 architecture but when trying to run it in jetson docker, am facing the same issue. Seems like the C files are not available in the docker.

Please let me know if you were able to find a workaround.

Thank you

marmikshah · November 25, 2022, 1:35am

Jetson docker does not have the libraries to build the static library. You will need to build it on the host by installing deepstream directly on host.

ahsanmuhammadakram1996 · November 29, 2022, 7:36am

Thank you so much for your kind response.

BTW I was able to run it on Jetson Docker.

alberto.ruiz · December 22, 2022, 6:23pm

From where did you get the files to compile?
Because from https://github.com/marcoslucianops/DeepStream-Yolo it does not work for me:

0:00:01.212635810 18436      0x1ccc720 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/watchbot/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/models/basic/yolov7n_b4_dla0_int8.engine open error
0:00:03.011958049 18436      0x1ccc720 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/models/basic/yolov7n_b4_dla0_int8.engine failed
0:00:03.012069058 18436      0x1ccc720 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/models/basic/yolov7n_b4_dla0_int8.engine failed, try rebuild
0:00:03.012111843 18436      0x1ccc720 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
YOLO config file or weights file is not specified

ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:03.012937387 18436      0x1ccc720 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 1]: build engine file failed
0:00:03.012992556 18436      0x1ccc720 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2020> [UID = 1]: build backend context failed
0:00:03.013025260 18436      0x1ccc720 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1257> [UID = 1]: generate backend failed, check config file settings
0:00:03.013085133 18436      0x1ccc720 WARN                 nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:03.013120909 18436      0x1ccc720 WARN                 nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary_yoloV7.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
0:00:03.013308847 18436      0x1ccc720 WARN                GST_PADS gstpad.c:1149:gst_pad_set_active:<primary_gie:sink> Failed to activate pad
** ERROR: <main:658>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance

I understand with this that it is required wts and cfg, not ONNX.

vcmike · December 22, 2022, 9:51pm

Hi,
We uploaded the code here:

comeon · April 6, 2023, 8:26am

hello, I couldn’t detect any targets, did you solve them later？

OforOmar · April 6, 2023, 11:20pm

I think the issue is caused because the exported model batch size > 1, it works with batch size =1 ,
for higher batch sizes I used this.

foreverneilyoung · March 26, 2024, 5:30pm

I’m having some doubts here, maybe somebody could jump in.

I’m running DS 6.0 on a Jetson Nano. Resnet10 and stuff works. Also yolo7tiny with an “inherited” setup.

Now I wanted to start from scratch and followed the advices given here.

While trying to run it I’m getting this, and I can’t figure out, what to do:

UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: DLA requests all profiles have same min, max, and opt value. All dla layers are falling back to GPU
ERROR: [TRT]: 4: [shapeCompiler.cpp::evaluateShapeChecks::832] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: reshape would change volume. IShuffleLayer /model/model.77/Reshape: reshaping failed for tensor: /model/model.77/m.0/Conv_output_0)
ERROR: Build engine failed from config file

Any help welcome

Topic		Replies	Views
ONNX as inference model (YoloV8) DeepStream SDK jetpack , onnx , deepstream	12	3339	July 10, 2023
Engine file and calib.table not saved in DeepStream DeepStream SDK tensorrt	6	881	December 25, 2022
Getting an error while trying to use deepstream_launchpad.ipynb code DeepStream SDK deepstream	28	1301	January 10, 2024
How to run deepstream-app without building engine? DeepStream SDK	6	473	March 11, 2024
Failed to parse ONNX model from file DeepStream SDK jetson , deepstream	4	23	January 15, 2025
Using custom model in deepstream DeepStream SDK jetson-inference , python , deepstream	40	74	September 10, 2024
Yolo type is not defined from config file name - Multistream DeepStream SDK	5	1716	October 12, 2021
Using trtexec to generate an engine file from an ONNX works error with two RTSP input source Jetson Orin Nano tensorrt	11	909	January 15, 2024
NvDsInfer Error: NVDSINFER_CONFIG_FAILED for FasterRCNN DeepStream SDK	13	635	November 22, 2023
Run BACK-TO-BACK-DETECTORS REFERENCE APP under DeepStream SDK 5.0 DeepStream SDK	16	994	October 12, 2021

Tutorial: How to run YOLOv7 on Deepstream

Related topics