DeepStream implementation of working nwesem/mtcnn_facenet_cpp_tensorRT needed

Deepstream 5.0
TensorRT 7.0
GPU:T4
CUDA 10.2
Hi
I have gone through a working TensorRT https://github.com/nwesem/mtcnn_facenet_cpp_tensorRT. I want to implement the same in deepstream. Can you kindly help.

Hi,

We have tested one of MTCNN model, and all layers are supported by the TensorRT.

$ /usr/src/tensorrt/bin/trtexec --deploy=./det1_relu.prototxt --output=conv4-2 --output=prob1

So the workflow should be simple.

1. Please modify from our /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_FasterRCNN/ sample.

2. Implement your bounding box based on the generateBbox.
Please wrap the bounding box below into the format as nvdsinfer_custom_impl_fasterRCNN.

3. Get your mean value based on this implementation.

4. Update config_infer_primary_fasterRCNN.txt file.

[property]
...
offsets=[B/G/R mean-subtraction value]
model-file=[your/model/name].caffemodel
proto-file=[your/proto/name].prototxt
labelfile-path=[your/label/name].txt
num-detected-classes=1
output-blob-names=conv4-2;prob1
parse-bbox-func-name=[your/bbox/function/name]
custom-lib-path=[your/bbox/library/path]
...

Thanks.

Hi I have tried to implement the same in nvdsparsebbox.cpp.But was not successful. Also can you share how this implementation can be worked upon if only one model is taken.

/*

  • Copyright © 2018-2019, NVIDIA CORPORATION. All rights reserved.
  • Permission is hereby granted, free of charge, to any person obtaining a
  • copy of this software and associated documentation files (the “Software”),
  • to deal in the Software without restriction, including without limitation
  • the rights to use, copy, modify, merge, publish, distribute, sublicense,
  • and/or sell copies of the Software, and to permit persons to whom the
  • Software is furnished to do so, subject to the following conditions:
  • The above copyright notice and this permission notice shall be included in
  • all copies or substantial portions of the Software.
  • THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  • IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  • FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
  • THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  • LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
  • FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
  • DEALINGS IN THE SOFTWARE.
    */

#include
#include
#include
#include “nvdsinfer_custom_impl.h”
#include “nvdssample_fasterRCNN_common.h”

#define MIN(a,b) ((a) < (b) ? (a) : (b))
#define MAX(a,b) ((a) > (b) ? (a) : (b))
#define CLIP(a,min,max) (MAX(MIN(a, max), min))

/* This is a sample bounding box parsing function for the sample FasterRCNN

  • detector model provided with the TensorRT samples. */

extern “C”
bool NvDsInferParseCustomFasterRCNN (std::vector const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
NvDsInferParseDetectionParams const &detectionParams,
std::vector &objectList);

/* C-linkage to prevent name-mangling */
extern “C”
bool NvDsInferParseCustomFasterRCNN (std::vector const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
NvDsInferParseDetectionParams const &detectionParams,
std::vector &objectList)
{
static int bboxPredLayerIndex = -1;
static int clsProbLayerIndex = -1;
static int roisLayerIndex = -1;
static const int NUM_CLASSES_FASTER_RCNN = 21;
static bool classMismatchWarn = false;
int numClassesToParse;

if (bboxPredLayerIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “bbox_pred”) == 0) {
bboxPredLayerIndex = i;
break;
}
}
if (bboxPredLayerIndex == -1) {
std::cerr << “Could not find bbox_pred layer buffer while parsing” << std::endl;
return false;
}
}

if (clsProbLayerIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “cls_prob”) == 0) {
clsProbLayerIndex = i;
break;
}
}
if (clsProbLayerIndex == -1) {
std::cerr << “Could not find cls_prob layer buffer while parsing” << std::endl;
return false;
}
}

if (roisLayerIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “rois”) == 0) {
roisLayerIndex = i;
break;
}
}
if (roisLayerIndex == -1) {
std::cerr << “Could not find rois layer buffer while parsing” << std::endl;
return false;
}
}

if (!classMismatchWarn) {
if (NUM_CLASSES_FASTER_RCNN !=
detectionParams.numClassesConfigured) {
std::cerr << “WARNING: Num classes mismatch. Configured:” <<
detectionParams.numClassesConfigured << ", detected by network: " <<
NUM_CLASSES_FASTER_RCNN << std::endl;
}
classMismatchWarn = true;
}

numClassesToParse = MIN (NUM_CLASSES_FASTER_RCNN,
detectionParams.numClassesConfigured);

float *rois = (float *) outputLayersInfo[roisLayerIndex].buffer;
float *deltas = (float *) outputLayersInfo[bboxPredLayerIndex].buffer;
float *scores = (float *) outputLayersInfo[clsProbLayerIndex].buffer;

int stride = 2;
int cellsize = 12;
int count = 0;
//score p
mydataFmt *p = score->pdata + score->width * score->height;
mydataFmt *plocal = location->pdata;
struct Bbox bbox;
struct orderScore order;
for (int row = 0; row < score->height; row++) {
for (int col = 0; col < score->width; col++) {
if (*p > Pthreshold) {
bbox.score = *p;
order.score = *p;
order.oriOrder = count;
bbox.x1 = round((stride * row + 1) / scale);
bbox.y1 = round((stride * col + 1) / scale);
bbox.x2 = round((stride * row + 1 + cellsize) / scale);
bbox.y2 = round((stride * col + 1 + cellsize) / scale);
bbox.exist = true;
bbox.area = (bbox.x2 - bbox.x1) * (bbox.y2 - bbox.y1);
for (int channel = 0; channel < 4; channel++)
bbox.regreCoord[channel] = *(plocal + channel * location->width * location->height);
boundingBox_.push_back(bbox);
bboxScore_.push_back(order);
count++;
}
p++;
plocal++;
}
}
return true;
}

/* Check that the custom function has been defined correctly */
CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(NvDsInferParseCustomFasterRCNN);

Hi,

Could you share the error message you met with us?
Thanks.

Parsing error had come. I think it is due to my implementation I had done in previus post

Hi,

Would you mind to share the error log with us?
Thanks.

Hi @AastaLLL, Kindly find the error log
Error log
deepstream-app -c deepstream_app_config_fasterRCNN.txt
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.
WARNING: …/nvdsinfer/nvdsinfer_func_utils.cpp:34 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
0:00:01.818410865 27752 0x5567a78c6960 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1577> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_FasterRCNN/det1_relu1.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x96x128
1 OUTPUT kFLOAT conv4-2 4x43x59
2 OUTPUT kFLOAT prob1 2x43x59

0:00:01.818493980 27752 0x5567a78c6960 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1681> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_FasterRCNN/det1_relu1.engine
0:00:01.819205855 27752 0x5567a78c6960 INFO nvinfer gstnvinfer_impl.cpp:311:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_FasterRCNN/config_infer_primary_fasterRCNN.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

** INFO: <bus_callback:167>: Pipeline running

Could not find bbox_pred layer buffer while parsing
0:00:01.967867822 27752 0x556798c486d0 ERROR nvinfer gstnvinfer.cpp:596:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::fillDetectionOutput() <nvdsinfer_context_impl_output_parsing.cpp:564> [UID = 1]: Failed to parse bboxes using custom parse function
Segmentation fault (core dumped)

Hi,

Please remember update the output layer name used in the customized parser:

if (strcmp(outputLayersInfo[i].layerName, “bbox_pred”) == 0) {
    bboxPredLayerIndex = i;
    break;
}

Suppose the bounding box layer of your model should be conv4-2.
Thanks.