Unable to update Deepstream default structure NvDsInferObjectDetectionInfo in file nvdsinfer.h

Hello,
We are currently running the custom deepstream mask rcnn app. Right now we are getting the deepstream default metadata which comprises of the bounding box information, frame id, timestamp etc but we also require the masks for each detection along with the current information.

As of now we are utilizing the custom mask rcnn app given on the nvidia IOT github and modifying the code in the nvdsinfer_custombboxparser_mrcnn_uff.cpp file. However we are able to access the masks in this file but we are unable to pass this mask to the main deepstream application.

One of the ways we tried was to modify the NvDsInferObjectDetectionInfo structure in the nvsdinfer.h file was by adding an additional parameter to store the masks information. The code compiles fine however when we run the deepstream application it gives us segmentation fault.

In the nvdsinfer.h file following changes were made:

typedef struct
{
  /** ID of the class to which the object belongs. */
  unsigned int classId;
  /** Horizontal offset of the bounding box shape for the object. */
  unsigned int left;
  /** Vertical offset of the bounding box shape for the object. */
  unsigned int top;
  /** Width of the bounding box shape for the object. */
  unsigned int width;
  /** Height of the bounding box shape for the object. */
  unsigned int height;
  

  [b]/** Test Variable */ 
  float mask[28 * 2 * 28 * 2];[/b]

  /** Object detection confidence. Should be a float value in the range [0,1] */
  float detectionConfidence;
} NvDsInferObjectDetectionInfo;

Is there any way we can make modifications in the above mentioned structures to get masks information along with the original metadata.

1 Like

Hi,

After recompiling, you also need to update the deepstream app which is located at /usr/bin.

There is an automatical script called install.sh in our .tbz2 package.
You can use it to update all the Deepstream app directly.

sudo ./install.sh
sudo ldconfig

The .tbz2 file can be found here:
https://developer.nvidia.com/deepstream-402-jetson

Thanks.

1 Like

Just to clarify we are using Tesla v100 Gpus’s will this file be compatible with it. Also we are updating the deepstream-app that we using by make clean and then make it again after the changes are made.

1 Like

Hi,

Here is a link for the desktop user:
https://developer.nvidia.com/deepstream-402-tesla

This script will help you update the binary located at /usr/bin/.

Thanks.

1 Like

Hello,

I am trying to get mask from Mask RCNN, I modified NvDsInferObjectDetectionInfo structure in the nvsdinfer.h file as the first comment. I also changed something in the nvdsinfer_custombboxparser_mrcnn_uff.cpp file. I created successfully .so file, but when I run on Deepstream, it’s fail. I think my nvdsinfer_custombboxparser_mrcnn_uff.cpp file have some problem. How can I fix that?

nvdsinfer_custombboxparser_mrcnn_uff.cpp

#include <cstring>
#include <iostream>
#include <vector>

#include <fstream>

#include "nvdsinfer_custom_impl.h"
#include <cassert>

#define MIN(a,b) ((a) < (b) ? (a) : (b))
#define MAX(a,b) ((a) > (b) ? (a) : (b))
#define CLIP(a,min,max) (MAX(MIN(a, max), min))
#define DIVIDE_AND_ROUND_UP(a, b) ((a + b - 1) / b)

using namespace std;

static const int DETECTION_MAX_INSTANCES = 100;

static const int NUM_CLASSES = 1 + 1; // COCO has 80 classes

static const int MASK_POOL_SIZE = 14;
static const nvinfer1::DimsCHW INPUT_SHAPE{3, 1024, 1024};

struct MRCNNBBox {
    float x1, y1, x2, y2;
};

struct MRCNNMask {
    float raw[MASK_POOL_SIZE * 2 * MASK_POOL_SIZE * 2];
};

struct MRCNNBBoxInfo {
    MRCNNBBox box;
    int label = -1;
    float prob = 0.0f;

    MRCNNMask* mask = nullptr;
};

struct RawDetection {
    float y1, x1, y2, x2, class_id, score;
};


extern "C"
bool NvDsInferParseCustomMrcnnUff (
    std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
    NvDsInferNetworkInfo  const &networkInfo,
    NvDsInferParseDetectionParams const &detectionParams,
    std::vector<NvDsInferObjectDetectionInfo> &objectList);

std::vector<MRCNNBBoxInfo> decodeOutput(void* detectionsHost, void* masksHost) {
    int input_dim_h = INPUT_SHAPE.d[1];
    int input_dim_w = INPUT_SHAPE.d[2];
    assert(input_dim_h == input_dim_w);

    std::vector<MRCNNBBoxInfo> binfo;


    RawDetection* detections = reinterpret_cast<RawDetection*>((float*) detectionsHost);
    MRCNNMask* masks = reinterpret_cast<MRCNNMask*>((float*) masksHost);


    for (int det_id = 0; det_id < DETECTION_MAX_INSTANCES; det_id++) {
        RawDetection cur_det = detections[det_id];
        MRCNNMask cur_mask = masks[det_id];

        int label = (int) cur_det.class_id;
        if (label <= 0)
            continue;
        
        MRCNNBBoxInfo det;
        det.label = label;
        det.prob = cur_det.score;

        det.box.x1 = cur_det.x1 ;
        det.box.y1 = cur_det.y1 ;
        det.box.x2 = cur_det.x2 ;
        det.box.y2 = cur_det.y2 ;

        if (det.box.x2 <= det.box.x1 || det.box.y2 <= det.box.y1)
            continue;

        det.mask = &cur_mask;
        binfo.push_back(det);

    }

    return binfo;
}

extern "C"
bool NvDsInferParseCustomMrcnnUff (std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                                   NvDsInferNetworkInfo  const &networkInfo,
                                   NvDsInferParseDetectionParams const &detectionParams,
                                   std::vector<NvDsInferObjectDetectionInfo> &objectList) {

    static int detIndex = -1;
    static int maskIndex = -1;

    if (detIndex == -1) {
        for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
            if (strcmp(outputLayersInfo[i].layerName, "mrcnn_detection") == 0) {
                detIndex = i;
                break;
            }
        }
        if (detIndex == -1) {
            std::cerr << "Could not find detection layer buffer while parsing" << std::endl;
            return false;
        }
    }

    if (maskIndex == -1) {
        for (unsigned int i = 0; i < outputLayersInfo.size(); i++) {
            if (strcmp(outputLayersInfo[i].layerName, "mrcnn_mask/Sigmoid") == 0) {
                maskIndex = i;
                break;
            }
        }
        if (maskIndex == -1) {
            std::cerr << "Could not find mask layer buffer while parsing" << std::endl;
            return false;
        }
    }

    float* out_det = (float *) outputLayersInfo[detIndex].buffer;
    float* out_mask = (float *) outputLayersInfo[maskIndex].buffer;

    std::vector<MRCNNBBoxInfo> binfo = decodeOutput(out_det, out_mask);

    for (unsigned int roi_id = 0; roi_id < binfo.size(); roi_id++) {
        NvDsInferObjectDetectionInfo object;
        object.classId = binfo[roi_id].label;
        object.detectionConfidence = binfo[roi_id].prob;

        object.left = CLIP(binfo[roi_id].box.x1 * networkInfo.width, 0, networkInfo.width - 1);
        object.top = CLIP(binfo[roi_id].box.y1 * networkInfo.height, 0, networkInfo.height - 1);
        object.width = CLIP((binfo[roi_id].box.x2 - binfo[roi_id].box.x1) * networkInfo.width, 0, networkInfo.width - 1);
        object.height = CLIP((binfo[roi_id].box.y2 - binfo[roi_id].box.y1) * networkInfo.height, 0, networkInfo.height - 1);

        for(int raw_id=0; raw_id < MASK_POOL_SIZE * 2 * MASK_POOL_SIZE * 2; raw_id++){
            object.mask_cus[raw_id] = binfo[roi_id].mask->raw[raw_id];

        objectList.push_back(object);
        
        }

    }

    return true;
}

CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(NvDsInferParseCustomMrcnnUff);

Thanks !

Hi nguyenanhquyethust,

Please open a new topic for your issue. Thanks

1 Like

Hi kayccc,

I sloved it on other topic.

Thanks.