I am trying to build a face recognition pipeline with deepstream. For detection, I am using redaction-app with the help of this article redaction-retinanet but I am not able to handle landmarks point. If i try to add any parameter to NvDsInferObjectDetectionInfo(nvdsinfer.h) i am getting segmentation fault.
Are you running a python app?
No i am trying to run deepstream app. Its almost similar to post mentioned. Just instead of retinanet i am using some other model in which in output layer i have bounding box and landmarks point. I can run without landmark.
It should work, could you refer the change in nvinfer is not populating 'confidence' field in NvDsObjectMeta (DS 4.0) - #20 by CJR firstly, and share your code with us if you still have issues.
My code does not reach to the callback function which deepstream passe to gstream pipeline. If i add any parameter to NvDsInferObjectDetectionInfo.I get segmentation fault. I am getting confidence value . My aim is to associate landmark array or 10 int/float value to each box.
Which callback? Are you referring the post process parser? Could you share your code with us?
Yes post process parser. My parser runs ok. Yes i can share You want to check parser code or Nvdsinfer code?
Share all what you had changed, I will try to setup locally, you can send me a private msg if you have any concern.
i am not able to upload code file here is I have changed to nvdsinfer.h
/**
Holds information about one parsed object from detector’s output.*/
typedef struct
{
/* ID of the class to which the object belongs. */
unsigned int classId;/** Horizontal offset of the bounding box shape for the object. */
unsigned int left;
/** Vertical offset of the bounding box shape for the object. */
unsigned int top;
/** Width of the bounding box shape for the object. */
unsigned int width;
/** Height of the bounding box shape for the object. */
unsigned int height;/** Object detection confidence. Should be a float value in the range [0,1] */
float detectionConfidence;/*** face landmark points holder */
std::vector landmarks ;
// unsigned int landmarks_points [10];} NvDsInferObjectDetectionInfo;
Thanks, Where did you access the landmarks filed, we cannot help to debug if you don’t share me with your change.
Also you can check the backtrace of core dump file to check where the app crash?
I was using it in my custom parser function. But there is not any difference even if i comment that code.
I gets segemtation fault after my custom parser return and before post parser callback. I can get “All Done” log.
Here is my custom parser code.
/**
- Copyright (c) 2017-2018, NVIDIA CORPORATION. All rights reserved.
- NVIDIA Corporation and its licensors retain all intellectual property
- and proprietary rights in and to this software, related documentation
- and any modifications thereto. Any use, reproduction, disclosure or
- distridbution of this software and related documentation without an express
- license agreement from NVIDIA Corporation is strictly prohibited.
*/
include
include
include
include “nvdsinfer_custom_impl.h”
include <math.h>define MIN(a,b) ((a) < (b) ? (a) : (b))
define MAX(a,b) ((a) > (b) ? (a) : (b))static float overlap1D(float x1min, float x1max, float x2min, float x2max)
{
if (x1min > x2min)
{
std::swap(x1min, x2min);
std::swap(x1max, x2max);
}
return x1max < x2min ? 0 : std::min(x1max, x2max) - x2min;
}static float computeIoU(const NvDsInferParseObjectInfo& bbox1, const NvDsInferParseObjectInfo& bbox2)
{
float overlapX
= overlap1D(bbox1.left, bbox1.left + bbox1.width, bbox2.left, bbox2.left + bbox2.width);
float overlapY
= overlap1D(bbox1.top, bbox1.top + bbox1.height, bbox2.top, bbox2.top + bbox2.height);
float area1 = (bbox1.width) * (bbox1.height);
float area2 = (bbox2.width) * (bbox2.height);
float overlap2D = overlapX * overlapY;
float u = area1 + area2 - overlap2D;
return u == 0 ? 0 : overlap2D / u;
}static bool compareBBoxConfidence(const NvDsInferParseObjectInfo& bbox1, const NvDsInferParseObjectInfo& bbox2)
{
return bbox1.detectionConfidence > bbox2.detectionConfidence;
}static std::vector
nonMaximumSuppression(std::vector inputBBoxInfo, const float nmsThresh)
{
std::stable_sort(inputBBoxInfo.begin(), inputBBoxInfo.end(), compareBBoxConfidence);
std::vector outputBBoxInfo;for (auto bbox1 : inputBBoxInfo) { bool keep = true; for (auto bbox2 : outputBBoxInfo) { if (keep) { float overlap = computeIoU(bbox1, bbox2); keep = overlap <= nmsThresh; } else break; } if (keep) outputBBoxInfo.push_back(bbox1); } return outputBBoxInfo;}
static std::vector
decodeCenterFaceTensor(
const float* heatmap, const float* scale, const float* offset, const float* landmarks, const uint& netW, const uint& netH,
const uint& W, const uint& H, const float probThresh)
{
std::vector bboxInfo;
for (unsigned int i = 0; i < H; i++) {
for (unsigned int j = 0; j < W; j++) {
int indx = iW + j;
int indx_ = indx + HW;
if (heatmap[indx]> probThresh)
{
float s0 = std::exp(scale[indx]);
float s1 = std::exp(scale[indx_]);
float o0 = std::exp(offset[indx]);
float o1 = std::exp(offset[indx_]);
s0*=4.0;
s1*=4.0;
float s = heatmap[indx];
float x1 = MAX(0, (j+o1+0.5)*4-s1/2);
float y1 = MAX(0, (i+o0+0.5)*4-s0/2);
x1 = MIN(x1, netW);
y1 = MIN(y1, netH);
float x2 = MIN(x1+s1, netW);
float y2 = MIN(y1+s0, netH);
NvDsInferParseObjectInfo object;
object.classId = 0;
object.detectionConfidence = s;
object.x1 = x1;
object.left = x1;
object.top = y1;
object.width = x2 - x1;
object.height = y2 - y1;
for (unsigned int k = 0; k < 5; k++){
object.landmarks.push_back((unsigned int)(x1+x2)/2;)
}
for (unsigned int k = 0; k < 5; k++){
object.landmarks.push_back((unsigned int)(y1+y2)/2;)
}
// for (unsigned int k = 0; k < 5; k++){
// // std::cout << k << " k " << std::endl;
// object.landmarks[k] = (unsigned int)(x1+x2)/2;
// }
// for (unsigned int k = 5; k < 10; k++){
// // std::cout << k << " k " << std::endl;
// object.landmarks[k] = ((unsigned int)(y1+y2)/2);
// }
// object.landmarks = landmarks_points;
// std::cout << x1 << " " << y1 << " " << x2 << " " << y2 << std::endl;bboxInfo.push_back(object); } } } // std::cout << " NMS Start " << std::endl; bboxInfo = nonMaximumSuppression(bboxInfo, 0.1); // std::cout << " NMS Done " << std::endl; return bboxInfo;}
/* This is a sample bounding box parsing function for the sample Resnet10
- detector model provided with the SDK. */
/* C-linkage to prevent name-mangling */
extern “C”
bool NvDsInferParseRetinaFace (std::vector const &outputLayersInfo,
NvDsInferNetworkInfo const &networkInfo,
NvDsInferParseDetectionParams const &detectionParams,
std::vector &objectList)
{
static int heatmapIndex = -1;
static int scaleIndex = -1;
static int offsetIndex = -1;
static int landmarksIndex = -1;
static NvDsInferDimsCHW heatmapLayerDims;
static NvDsInferDimsCHW scaleLayerDims;
static NvDsInferDimsCHW offsetLayerDims;
static NvDsInferDimsCHW landmarksLayerDims;
int numDetsToParse;/* Find the bbox layer */
if (heatmapIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “537”) == 0) {
heatmapIndex = i;
getDimsCHWFromDims(heatmapLayerDims, outputLayersInfo[i].dims);
break;
}
std::cout << outputLayersInfo[i].layerName << “Gulshan” << i << std::endl;
}
if (heatmapIndex == -1) {
std::cerr << “Could not find heatmap layer buffer while parsing” << std::endl;
return false;
}
}/* Find the scores layer */
if (scaleIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “538”) == 0) {
scaleIndex = i;
getDimsCHWFromDims(scaleLayerDims, outputLayersInfo[i].dims);
break;
}
}
if (scaleIndex == -1) {
std::cerr << “Could not find scale layer buffer while parsing” << std::endl;
return false;
}
}/* Find the classes layer */
if (offsetIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “539”) == 0) {
offsetIndex = i;
break;
}
}
if (offsetIndex == -1) {
std::cerr << “Could not find offset layer buffer while parsing” << std::endl;
return false;
}
}/* Find the classes layer */
if (landmarksIndex == -1) {
for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
if (strcmp(outputLayersInfo[i].layerName, “539”) == 0) {
landmarksIndex = i;
break;
}
}
if (landmarksIndex == -1) {
std::cerr << “Could not find landmarks layer buffer while parsing” << std::endl;
return false;
}
}/* Calculate the number of detections to parse */
numDetsToParse = heatmapLayerDims.c;float *heatmap = (float *) outputLayersInfo[heatmapIndex].buffer;
float *scale = (float *) outputLayersInfo[scaleIndex].buffer;
float *offset = (float *) outputLayersInfo[offsetIndex].buffer;
float *landmarks = (float *) outputLayersInfo[landmarksIndex].buffer;std::vector objects = decodeCenterFaceTensor(heatmap, scale, offset, landmarks,
networkInfo.width, networkInfo.height, heatmapLayerDims.w, heatmapLayerDims.h, 0.35);objectList = objects;
if(objects.size() > 0){
std::cout << " All Done " << objects.size() << std::endl;
}
return true;
}/* Check that the custom function has been defined correctly */
CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(NvDsInferParseRetinaFace);
Hey Customer,
Would you mind to share me the source file which can be built well, there are many build error using your code to build the parser lib. It’s really difficult to fix the build error.
in addition,
- have you checked the backtrace of the core dump file?
- you add landmarks filed in the struct, I think its type should be
/*** face landmark points holder */
std::vector landmarks ; ==>std::vector<unsigned int> landmarks ; // right?
sorry that was a typo mistake (landmark type).
how can I check the backtrace of the core dump file?
I don’t have any problem with sharing anything but actually I have upgraded to Cuda 10.2 for which deepstream is not available and I have made a lot of changes to many related files/libraries(Tensorrt etc…). I will have to go back to Cuda 10.1 and rebuild it again. I will do that later. If you want to reproduce the same error just change NvDsInferObjectDetectionInfo in this. Just add any float, int anything you will get the error. I can share the config file, parser file, engine file.
Actually, I had added a filed in NvDsInferObjectDetectionInfo like what you did and access the filed in post process parser and it can work well. So I guess there must be something wrong in your code or setup. For how to check backtrace of the core dump file, pls refer linux - How do I analyze a program's core dump file with GDB when it has command-line parameters? - Stack Overflow .
Anyway, you can share all what you changed with us for further debug, the nvdsinfer.h, post process parser code which can be built, and the config, engine file if you would like.
Thanks I will try to debug and will share whth you all things.