vcmike
September 27, 2022, 5:15am
1
Hi,
To help people run official YOLOv7 models on Deepstream here is some helper code.
You should first export the model to ONNX via this command (taken from the yolov7 README)
python export.py --weights ./yolov7-tiny.pt --grid --end2end --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640
This command will create an ONNX model with an efficientNMS node. To be able to map the model outputs from efficientNMS to the NVIDIA NvDsObjectMeta data structure you need this code which needs to be added/compiled in /opt/nvidia/deepstream/deepstream-6.1/sources/libs/nvdsinfer_customparser/nvdsinfer_custombboxparser.cpp
extern "C"
bool NvDsInferParseCustomEfficientNMS (std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
NvDsInferParseDetectionParams const &detectionParams,
std::vector<NvDsInferObjectDetectionInfo> &objectList) {
if(outputLayersInfo.size() != 4)
{
std::cerr << "Mismatch in the number of output buffers."
<< "Expected 4 output buffers, detected in the network :"
<< outputLayersInfo.size() << std::endl;
return false;
}
const char* log_enable = std::getenv("ENABLE_DEBUG");
int* p_keep_count = (int *) outputLayersInfo[0].buffer;
float* p_bboxes = (float *) outputLayersInfo[1].buffer;
NvDsInferDims inferDims_p_bboxes = outputLayersInfo[1].inferDims;
int numElements_p_bboxes=inferDims_p_bboxes.numElements;
float* p_scores = (float *) outputLayersInfo[2].buffer;
unsigned int* p_classes = (unsigned int *) outputLayersInfo[3].buffer;
const float threshold = detectionParams.perClassThreshold[0];
float max_bbox=0;
for (int i=0; i < numElements_p_bboxes; i++)
{
if ( max_bbox < p_bboxes[i] )
max_bbox=p_bboxes[i];
}
if (p_keep_count[0] > 0)
{
assert (!(max_bbox < 2.0));
for (int i = 0; i < p_keep_count[0]; i++) {
if ( p_scores[i] < threshold) continue;
assert((unsigned int) p_classes[i] < detectionParams.numClassesConfigured);
NvDsInferObjectDetectionInfo object;
object.classId = (int) p_classes[i];
object.detectionConfidence = p_scores[i];
object.left=p_bboxes[4*i];
object.top=p_bboxes[4*i+1];
object.width=(p_bboxes[4*i+2] - object.left);
object.height= (p_bboxes[4*i+3] - object.top);
if(log_enable != NULL && std::stoi(log_enable)) {
std::cout << "label/conf/ x/y w/h -- "
<< p_classes[i] << " "
<< p_scores[i] << " "
<< object.left << " " << object.top << " " << object.width << " "<< object.height << " "
<< std::endl;
}
object.left=CLIP(object.left, 0, networkInfo.width - 1);
object.top=CLIP(object.top, 0, networkInfo.height - 1);
object.width=CLIP(object.width, 0, networkInfo.width - 1);
object.height=CLIP(object.height, 0, networkInfo.height - 1);
objectList.push_back(object);
}
}
return true;
}
You then just need to make sure your nvinfer stages point to the correct custom library and function like:
parse-bbox-func-name=NvDsInferParseCustomEfficientNMS
custom-lib-path=/opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infercustomparser.so
With that you should be good to go.
3 Likes
Hello, could you help me please. I run this example deepstream_python_apps/apps/deepstream-rtsp-in-rtsp-out at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub
python3: nvdsinfer_custombboxparser.cpp:589: bool NvDsInferParseCustomEfficientNMS(const std::vector&, const NvDsInferNetworkInfo&, const NvDsInferParseDetectionParams&, std::vector&): Assertion `(unsigned int) p_classes[i] < detectionParams.numClassesConfigured’ failed.
img_log
vcmike
October 6, 2022, 9:27pm
4
Hi @xenonscrinium .
The issue is probably that you have not updated num-detected-classes=91 to be correct for the model you are running. This assertion checks that any detected class from the model is less than this num-detected-classes variable.
1 Like
zongxp
October 12, 2022, 10:13am
5
It works, thanks for your sharing.
Hello,
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=yolov7-tiny.onnx
model-engine-file=yolov7-tiny.onnx_b2_gpu0_fp16.engine
labelfile-path=models/coco.txt
num-detected-classes=80
operate-on-class-ids=0;1;2;9
interval=0
gie-unique-id=1
batch-size=2
network-mode=2
process-mode=1
network-type=2
cluster-mode=2
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomEfficientNMS
custom-lib-path=libnvds_infercustomparser.so
[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
topk=300
vcmike
November 10, 2022, 6:28am
7
operate-on-class-ids=0;1;2;9 is for secondary inference so might be breaking this?
Tried running without this line, nothing still :(
vcmike
November 10, 2022, 6:41am
9
you could try export ENABLE_DEBUG=1 as it will show if there are actually any detections
Anyone willing to share a gst-launch example to try this?
Nevermind, works flawlessly!
1 Like
When I run ‘sudo -E make install’ I get the following problems which don’t seem realted to the code posted above (Jeston Xavier NX):
g++ -o libnvds_infercustomparser.so nvdsinfer_custombboxparser.cpp nvdsinfer_customclassifierparser.cpp -Wall -std=c++11 -shared -fPIC -I…/…/includes -I /usr/local/cuda-11.4/include -Wl,–start-group -lnvinfer -lnvparsers -Wl,–end-group#include “crt/host_defines.h”#include “crt/host_defines.h”
Any idea?
1 Like
vcmike:
extern "C"
bool NvDsInferParseCustomEfficientNMS (std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
NvDsInferParseDetectionParams const &detectionParams,
std::vector<NvDsInferObjectDetectionInfo> &objectList) {
if(outputLayersInfo.size() != 4)
{
std::cerr << "Mismatch in the number of output buffers."
<< "Expected 4 output buffers, detected in the network :"
<< outputLayersInfo.size() << std::endl;
return false;
}
const char* log_enable = std::getenv("ENABLE_DEBUG");
int* p_keep_count = (int *) outputLayersInfo[0].buffer;
float* p_bboxes = (float *) outputLayersInfo[1].buffer;
NvDsInferDims inferDims_p_bboxes = outputLayersInfo[1].inferDims;
int numElements_p_bboxes=inferDims_p_bboxes.numElements;
float* p_scores = (float *) outputLayersInfo[2].buffer;
unsigned int* p_classes = (unsigned int *) outputLayersInfo[3].buffer;
const float threshold = detectionParams.perClassThreshold[0];
float max_bbox=0;
for (int i=0; i < numElements_p_bboxes; i++)
{
if ( max_bbox < p_bboxes[i] )
max_bbox=p_bboxes[i];
}
if (p_keep_count[0] > 0)
{
assert (!(max_bbox < 2.0));
for (int i = 0; i < p_keep_count[0]; i++) {
if ( p_scores[i] < threshold) continue;
assert((unsigned int) p_classes[i] < detectionParams.numClassesConfigured);
NvDsInferObjectDetectionInfo object;
object.classId = (int) p_classes[i];
object.detectionConfidence = p_scores[i];
object.left=p_bboxes[4*i];
object.top=p_bboxes[4*i+1];
object.width=(p_bboxes[4*i+2] - object.left);
object.height= (p_bboxes[4*i+3] - object.top);
if(log_enable != NULL && std::stoi(log_enable)) {
std::cout << "label/conf/ x/y w/h -- "
<< p_classes[i] << " "
<< p_scores[i] << " "
<< object.left << " " << object.top << " " << object.width << " "<< object.height << " "
<< std::endl;
}
object.left=CLIP(object.left, 0, networkInfo.width - 1);
object.top=CLIP(object.top, 0, networkInfo.height - 1);
object.width=CLIP(object.width, 0, networkInfo.width - 1);
object.height=CLIP(object.height, 0, networkInfo.height - 1);
objectList.push_back(object);
}
}
return true;
}
I was able to run the model using the provided tutorial on x86 architecture but when trying to run it in jetson docker, am facing the same issue. Seems like the C files are not available in the docker.
Please let me know if you were able to find a workaround.
Thank you
Jetson docker does not have the libraries to build the static library. You will need to build it on the host by installing deepstream directly on host.
Thank you so much for your kind response.
BTW I was able to run it on Jetson Docker.
From where did you get the files to compile?https://github.com/marcoslucianops/DeepStream-Yolo it does not work for me:
0:00:01.212635810 18436 0x1ccc720 WARN omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/watchbot/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/models/basic/yolov7n_b4_dla0_int8.engine open error
0:00:03.011958049 18436 0x1ccc720 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/models/basic/yolov7n_b4_dla0_int8.engine failed
0:00:03.012069058 18436 0x1ccc720 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/models/basic/yolov7n_b4_dla0_int8.engine failed, try rebuild
0:00:03.012111843 18436 0x1ccc720 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
YOLO config file or weights file is not specified
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:03.012937387 18436 0x1ccc720 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 1]: build engine file failed
0:00:03.012992556 18436 0x1ccc720 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2020> [UID = 1]: build backend context failed
0:00:03.013025260 18436 0x1ccc720 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1257> [UID = 1]: generate backend failed, check config file settings
0:00:03.013085133 18436 0x1ccc720 WARN nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:03.013120909 18436 0x1ccc720 WARN nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary_yoloV7.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
0:00:03.013308847 18436 0x1ccc720 WARN GST_PADS gstpad.c:1149:gst_pad_set_active:<primary_gie:sink> Failed to activate pad
** ERROR: <main:658>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance
I understand with this that it is required wts and cfg, not ONNX.
vcmike
December 22, 2022, 9:51pm
18
Hi,
