Accessing segmentation metadata for Maskrcnn

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi,

I am trying to access the output masks of my maskrcnn model as numpy arrays following the deepstream-segmentation python example, but the frame_meta.frame_user_meta_list is always None.
I am using the new version output parser for the maskrcnn NvDsInferParseCustomMrcnnTLTV2 and this is my config file
config_infer_primary_mrcnn.txt (2.2 KB)

Any help would be appreciated, thank you.

**• Hardware Platform ** Jetson AGX Xavier
• DeepStream Version deepstream 5.1
• JetPack Version (valid for Jetson only) Jetpack 4.5.1
• TensorRT Version Tensorrt 7.1.3

3

Can you run the model well using GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

4

Yes, i also have to clarify that in my case described above, even though frame_meta.frame_user_meta_list is none, the model does output masks and detections and they are visible on the output frames, so my model works well, i just can’t access the metadata.

5

Any updates regarding this point?

6

Where did you access the meta data?

7

I placed the probe that is attempting to access the metadata after the nvosd plugin.

8

It’s expected since the meta data will be released after nvosd, can you install the probe on the sink of nvosd or any upstream plugin to see if you can access it?

9

I tried installing the probe on the src pad of the Nvinfer plugin as well as the sink pad of the tiler plugin, i still can’t access it.

10

Can you try the C version example on my first example and then export it to python?
Also keep in mind mask rcnn is instance segmentation while the python segmentation is semantic segmentation, you should not expect the python sample work since they are different types of models

11

I did try the C version examples, they work fine with my model as a pipeline but i still can’t access the meta data even in the C examples.
I know instance segmentation is different but i assumed that the segmentation meta data whether it’s semantic or instance would still be stored in the frame_meta.frame_user_meta_list.
If that’s not the case, then where should i be looking for the masks metadata for instance segmentation models?

12

No, that’s not true, instance segmentation model will store the mask info

  /** Holds mask parameters for the object. This mask is overlayed on object
   * @see NvOSD_MaskParams. */
  NvOSD_MaskParams mask_params;
13

Tried that one too in python, i got the following error

TypeError: Unable to convert function return value to a Python type! The signature was
	(self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams

Does this mean that the Mask metadata cannot be accessed from python?

Update:
I tried to access the mask info in C, i was able to access it, but i am not sure i understand the values or if these are the values i am looking for. According to the NvOSD_MaskParams documentation, the available data are just the size, threshold, width and height of the mask not the actual mask, Unless i am missing something.

14

Could you create a new topic for the document issue, there is a data filed inside the struct if you checked the header file

15

what about the python bindings? Are they not implemented?