How to get the amount of pixels belonging to Mask RCNN mask

[aurelm95]

How to get the amount of pixels belonging to Mask RCNN mask

• Hardware Platform (Jetson / GPU) Jetson nano

• DeepStream Version 6.0

• JetPack Version (valid for Jetson only) 4.6

• TensorRT Version 8.0.1

I want to count the numer of pixels belonging to each segmented instance using Mask RCNN.

To do so, first I wanted to acces to the mask data in the same way I can acces to the detection data when I use a detection model.

From the documentation: NVIDIA DeepStream SDK API Reference: nvll_osd_struct.h Source File

I see that can acces to its values with the following lines:

NvOSD_MaskParams *mask = (NvOSD_MaskParams*) &(obj_meta->mask_params);
g_print("Aure: mask: size: %u, threshold: %f, width: %u, height: %u\n",mask->size,mask->threshold,mask->width,mask->height);

However, I allways see the same following log FOR EVERY INSTACE in EVERY FRAME:

Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Frame Number = 38 Vehicle Count = 3 Person Count = 0
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Frame Number = 39 Vehicle Count = 4 Person Count = 0
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28
Aure: mask: size: 3136, threshold: 0.000000, width: 28, height: 28

which make no sense.

Note: I can perfectly visualize the masks and its corresponding bboxes and class, so the model is doing fine.

What I am doing wrong ? How should I acces to the mask data?

Thanks you in Advance!

[fanzh]

I will check.

[aurelm95]

I have found the following function in /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-mrcnn-test/deepstream_mrcnn_test.cpp

static GList *
generate_mask_polygon (float *mask, int mask_width, int mask_height,
                                    int rect_width, int rect_height,
                                    float threshold)
{
GList *mask_list = NULL;
vector<vector<cv::Point> > contours;
cv::Mat dst = cv::Mat(rect_height, rect_width, CV_8UC1);

resizeMask(mask, mask_width, mask_height, dst, threshold);

findContours (dst, contours, RETR_TREE, CHAIN_APPROX_SIMPLE);

for( size_t i = 0; i< contours.size(); i++ ) {
    GArray *polygon;
    vector<cv::Point> temp = contours[i];
    polygon = g_array_new (FALSE, FALSE, sizeof(gdouble));

    for (size_t j = 0; j < temp.size(); j++) {
    gdouble xx = (gdouble)temp[j].x;
    gdouble yy = (gdouble)temp[j].y;
    g_array_append_val (polygon, xx);
    g_array_append_val (polygon, yy);
    }
    mask_list = g_list_append (mask_list, polygon);
}
return mask_list;
}

which, apparently, works fine. Now I onle need to find an efficient way to count all the points inside the polygon :D

By the way, the python bindings (pyds) does not have the functions resizeMask, findContours or generate_mask_polygon right?

[fanzh]

yes, pyds is used to convert dessptream struct and function.

[aurelm95]

yes, I know, but there is not such a function like pyds.generate_mask_polygon right?

[fanzh]

yes, you can find generate_mask_polygon definition in deepstream\deepstream\sources\apps\sample_apps\deepstream-mrcnn-test\deepstream_mrcnn_test.cpp, it is not a deepstream function, so there is not such a function like pyds.generate_mask_polygon.