# Get mask data from \`nvdsinfer\_custombboxparser\_mrcnn\_uff\` mrcnn parser plugin

**URL:** <https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754>\
**Category:** DeepStream SDK\
**Tags:** gstreamer\
**Created:** [January 17, 2022, 10:51am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754 "2022-01-17T10:51:01Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![pradan](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradan/32/133970_2.png) [@pradan](https://forums.developer.nvidia.com/u/pradan)\
**Post date:** [January 17, 2022, 10:51am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/1 "2022-01-17T10:51:01Z")

</div>

**• Hardware Platform (Jetson / GPU)** Jetson Xavier AGX  
**• DeepStream Version** 5.1  
**• JetPack Version (valid for Jetson only)** 4.5.1  
**• TensorRT Version** 7.1.3  
**• NVIDIA GPU Driver Version (valid for GPU only)**  
**• Issue Type( questions, new requirements, bugs)** Question  
**• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)**

I am running an optimised version of Mask-RCNN by obtaining Uff file using TensorRT which I used to obtain an `engine` file. I run inference on this using the [Deepstream4.x samples](https://github.com/NVIDIA-AI-IOT/deepstream_4.x_apps) which provides a custom parser for mrcnn: [nvdsinfer\_customparser\_mrcnn\_uff](https://github.com/NVIDIA-AI-IOT/deepstream_4.x_apps/tree/master/nvdsinfer_customparser_mrcnn_uff). The compiled app provides only bounding boxes as output (overlayed on the frame) but I require the mask too.

Some users suggest [dumping the `mask_data` within the parser](https://github.com/NVIDIA-AI-IOT/deepstream_4.x_apps/issues/10#issuecomment-603748186) but this doesn’t solve the purpose for those avoiding writing files during operation. It also requires scaling the mask. I noticed the custom parser to be directly connected to `nvtiler`:

 ![gstreamer pipline snap](https://global.discourse-cdn.com/nvidia/original/3X/9/a/9a4c31b3bc436e2098c3d0d87f8a90021a2992b9.jpeg)

`nvtiler` (defined in [`gst-nvmultistreamtiler`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmultistreamtiler.html#gst-nvmultistreamtiler)) performs all the re-scaling and post-inference operations on the inference results (bounding box). The mask data is not forwarded to this element as I can see in the code snippet from the

```nohighlight
    for (unsigned int roi_id = 0; roi_id < binfo.size(); roi_id++) {
        NvDsInferObjectDetectionInfo object; // *****NOTICE THIS********
        object.classId = binfo[roi_id].label;
        object.detectionConfidence = binfo[roi_id].prob;

        /* Clip object box co-ordinates to network resolution */
        object.left = CLIP(binfo[roi_id].box.x1 * networkInfo.width, 0, networkInfo.width - 1);
        object.top = CLIP(binfo[roi_id].box.y1 * networkInfo.height, 0, networkInfo.height - 1);
        object.width = CLIP((binfo[roi_id].box.x2 - binfo[roi_id].box.x1) * networkInfo.width, 0, networkInfo.width - 1);
        object.height = CLIP((binfo[roi_id].box.y2 - binfo[roi_id].box.y1) * networkInfo.height, 0, networkInfo.height - 1);

        objectList.push_back(object);
    }

```

Although the docs for [gst-nvinfer](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer) do mention that its capable of doing so:

 ![Screenshot 2022-01-17 at 4.15.46 PM](https://global.discourse-cdn.com/nvidia/original/3X/a/2/a2661c8bd2384b42778a38c0c650029546b39d49.png)

But I can’t figure out how; I failed to find the definition of `NvDsInferObjectDetectionInfo` (the data structure used to accomodate the inference resutls) and only got the link to [nvdsinfer\_custom\_impl.h](https://docs.nvidia.com/metropolis/deepstream/4.0/dev-guide/DeepStream_Development_Guide/baggage/nvdsinfer __custom__ impl_8h_source.html)

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [January 18, 2022, 2:35am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/3 "2022-01-18T02:35:23Z")

</div>

The deepstream 4.x is not supported any longer. Please upgrade to latest DeepStream 6.0 GA.

The mask data is stored in [mask\_params](https://docs.nvidia.com/metropolis/deepstream/sdk-api/struct__NvDsObjectMeta.html#a5d5f0990943cc6368c2e584053e478a0) in [NvDsObjectMeta](https://docs.nvidia.com/metropolis/deepstream/sdk-api/struct__NvDsObjectMeta.html). Its is a [\_NvOSD\_MaskParams struct](https://docs.nvidia.com/metropolis/deepstream/sdk-api/struct __NvOSD__ MaskParams.html)

Please study the DeepStream MetaData by [MetaData in the DeepStream SDK — DeepStream 6.1.1 Release documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_metadata.html)

---

<div class="post-metadata">

**Author:** ![pradan](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradan/32/133970_2.png) [@pradan](https://forums.developer.nvidia.com/u/pradan)\
**Post date:** [January 19, 2022, 10:19am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/4 "2022-01-19T10:19:02Z")

</div>

Hi Fiona, I am using 5.1 and the 4.x repository is just used for the custom parser.

Thanks for the references. I tried to log the contents of the

```nohighlight
// Extract mask
            NvOSD_MaskParams maskParams = obj_meta->mask_params;
            float * maskParamsData = maskParams.data;
            unsigned int maskParamsSize = maskParams.size;
            unsigned int maskParamsWidth = maskParams.width;
            unsigned int maskParamsHeight = maskParams.height;
            // log
            g_print ("maskParamsSize = %d maskParamsWidth = %d "
              "maskParamsHeight = %d\n",
              maskParamsSize, maskParamsWidth, maskParamsHeight);

```

where `obj_meta` is defined as

```nohighlight
obj_meta = (NvDsObjectMeta *) (l_obj->data);

```

as per the `/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-image-decode-test` (the sample that I use currenlty).

But all values return 0. Also, By looking at the parser’s code, I don’t think the mask data is being exported outside the parser. Also, the gstreamer pipeline, when visualised, doesn’t make sense to be sending anything from `NvDsInferObjectDetectionInfo` directly to `GstNvDsOsd`.

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/c/4/c4c7dbc367335a15fcaa5eedf9fad36403c0bcf1.png)

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [January 20, 2022, 3:03am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/5 "2022-01-20T03:03:46Z")

</div>

Have you set “output-instance-mask=1” with nvinfer?

[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS\_plugin\_gst-nvinfer.html#id2](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#id2)

We have proper sample for TAO models. [NVIDIA-AI-IOT/deepstream\_tao\_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps).

---

<div class="post-metadata">

**Author:** ![pradan](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradan/32/133970_2.png) [@pradan](https://forums.developer.nvidia.com/u/pradan)\
**Post date:** [January 20, 2022, 11:21am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/6 "2022-01-20T11:21:13Z")

</div>

I have set the params as per the guide shared by you. It also replaces the `NvDsInferParseCustomMrcnnUff ` with `NvDsInferParseCustomMrcnnTLT`. As my model is not trained using TAO, I get errors despite successful compilation.

```nohighlight
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:07.963338528 9343 0x55b1fadb90 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 1]: deserialized trt engine from :/home/virus/Desktop/optimisation/deepstream_4.x_apps/res101-holygrail-ep26-fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_image 3x1024x1024     
1 OUTPUT kFLOAT mrcnn_detection 100x6           
2 OUTPUT kFLOAT mrcnn_mask/Sigmoid 100x4x28x28     

0:00:07.963583302 9343 0x55b1fadb90 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1806> [UID = 1]: Use deserialized engine model: /home/virus/Desktop/optimisation/deepstream_4.x_apps/res101-holygrail-ep26-fp16.engine
0:00:08.095824565 9343 0x55b1fadb90 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:pgie_mrcnn_uff_config.txt sucessfully
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:09.985972782 9343 0x55b19a3b70 ERROR nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::parseBoundingBox() <nvdsinfer_context_impl_output_parsing.cpp:59> [UID = 1]: Could not find output coverage layer for parsing objects
0:00:09.986096721 9343 0x55b19a3b70 ERROR nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::fillDetectionOutput() <nvdsinfer_context_impl_output_parsing.cpp:733> [UID = 1]: Failed to parse bboxes
Segmentation fault (core dumped)

```

This hints towards the fact that either I re-train using TAO or write/modify the `NvDsInferParseCustomMrcnnUff` (the parser that works fine but only gives me bounding box) such that it also transports the mask metadata to the parent program.

Am I correct in my evaluation ?

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [January 21, 2022, 9:22am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/7 "2022-01-21T09:22:24Z")

</div>

> [@pradan](#):
>
> This hints towards the fact that either I re-train using TAO or write/modify the `NvDsInferParseCustomMrcnnUff` (the parser that works fine but only gives me bounding box) such that it also transports the mask metadata to the parent program.

So please check with your network. The mrcnn post-processing is open source.  
[deepstream\_tao\_apps/post\_processor at release/tlt3.0 · NVIDIA-AI-IOT/deepstream\_tao\_apps (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/tree/release/tlt3.0/post_processor)

---

<div class="post-metadata">

**Author:** ![pradan](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradan/32/133970_2.png) [@pradan](https://forums.developer.nvidia.com/u/pradan)\
**Post date:** [January 25, 2022, 6:45am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/9 "2022-01-25T06:45:26Z")

</div>

Hi Fiona. I augmented the `NvDsInferParseCustomMrcnnUff` code such that the mask metadata is also returned along with the bbox. For this, I use `NvDsInferParseCustomMrcnnTLTV2` as reference. So both use different structures to return data

```auto
 NvDsInferParseCustomMrcnnUff : NvDsInferObjectDetectionInfo
 NvDsInferParseCustomMrcnnTLTV2 : NvDsInferInstanceMaskInfo

```

What I needed was a combination of both so I made some modifications and was able to

- infer bbox+mask data for each inferred object
- append them to the object list `objectList`
- Return this list in a container `vector<NvDsInferInstanceMaskInfo>`

On changing the config file to display masks (as per [your suggestion](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/5)), I get errors as

```console
oadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:pgie_mrcnn_uff_config.txt sucessfully
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:05.026531740 9152 0x558966e370 ERROR nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::parseBoundingBox() <nvdsinfer_context_impl_output_parsing.cpp:59> [UID = 1]: Could not find output coverage layer for parsing objects
0:00:05.026740390 9152 0x558966e370 ERROR nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::fillDetectionOutput() <nvdsinfer_context_impl_output_parsing.cpp:733> [UID = 1]: Failed to parse bboxes
Segmentation fault (core dumped)

```

I am aware that mixing parsers originally meant for TAO-based app with that meant for the old mrcnn samples is to be blamed here. So still, I wish to continue with this approach and derive/obtain the masks myself.

## So this is where I’ll need your help:

The mask+bbox data is returned to the next elements of the gsteramer/deepstream pipeline and the OSD element uses this data to overlay bbox on the displayed frame. Similar to this, I wish to play with the mask data but outside of the parser function (preferrably in the main app:`deepstream_custom.c`). Can you guide me in the right direction ?

\*P.S. I know I can look into the source of the OSD element and draw the mask there but this won’t be much helpful as I want to do other things with this mask data right in the main app: `deepstream_custom.c`. I am pretty sure there exists a method to access the metadata of the `nvinfer` plugin right, but just can’t find the entry to the rabbit hole \*

---

<div class="post-metadata">

**Author:** ![aurelm95](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aurelm95](https://forums.developer.nvidia.com/u/aurelm95)\
**Post date:** [September 15, 2022, 3:27pm UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/11 "2022-09-15T15:27:27Z")

</div>

> [@pradan](#):
>
> NvDsInferInstanceMaskInfo

Hello @pradan, I am tryin to do exactly the same as you. Have you solved the problem? Otherwise I will have to retrain with the tlt method :(

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [September 19, 2022, 2:26am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/12 "2022-09-19T02:26:39Z")

</div>

First, DeepStream 4.x samples are not supported now. Please update to latest DeepStream 6.1.1 and refer to the latest sample: [NVIDIA-AI-IOT/deepstream\_tao\_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps)

> [@pradan](#):
>
> I augmented the `NvDsInferParseCustomMrcnnUff` code such that the mask metadata is also returned along with the bbox.

You don’t need to do anything with the post-processing function. The mask and bbox have been output with NvDsInferInstanceMaskInfo already, please refer to NvDsInferParseCustomMrcnnTLTV2() in [deepstream\_tao\_apps/nvdsinfer\_custombboxparser\_tao.cpp at master · NVIDIA-AI-IOT/deepstream\_tao\_apps (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/blob/master/post_processor/nvdsinfer_custombboxparser_tao.cpp).

You can make a little change to the code of [deepstream\_tao\_apps/deepstream\_det\_app.c at master · NVIDIA-AI-IOT/deepstream\_tao\_apps (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/blob/master/apps/tao_detection/deepstream_det_app.c) to support display mask and bbox at the same time.

Please change the line [deepstream\_tao\_apps/deepstream\_det\_app.c at master · NVIDIA-AI-IOT/deepstream\_tao\_apps (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/blob/master/apps/tao_detection/deepstream_det_app.c#L776) as the following:  
`if(showMask)`

```
  `g_object_set (G_OBJECT (nvdsosd), "display-mask", 1, "display-bbox", 1, "display-text", 0, "process-mode", 0, NULL);`

```

And then build and run the sample of peopleSegNet(this is Mrcnn network) as the instruction [deepstream\_tao\_apps/README.md at master · NVIDIA-AI-IOT/deepstream\_tao\_apps (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/blob/master/README.md)

---

<div class="post-metadata">

**Author:** ![aurelm95](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aurelm95](https://forums.developer.nvidia.com/u/aurelm95)\
**Post date:** [September 19, 2022, 7:14am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/13 "2022-09-19T07:14:05Z")

</div>

Is there an option to run a mrcnn .uff model with deepstream 6.x ? or trainning with tao is a must??

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [September 19, 2022, 7:25am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/15 "2022-09-19T07:25:01Z")

</div>

> [@aurelm95](#):
>
> Is there an option to run a mrcnn .uff model with deepstream 6.x ? or trainning with tao is a must??

Please refer to your own topic  
[Displaying masks from a mrcnn .uff file? - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/displaying-masks-from-a-mrcnn-uff-file/228151)

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [October 3, 2022, 7:25am UTC](https://forums.developer.nvidia.com/t/get-mask-data-from-nvdsinfer-custombboxparser-mrcnn-uff-mrcnn-parser-plugin/200754/16 "2022-10-03T07:25:06Z")

</div>

Closed on October 3, 2022, 7:25am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.