Error when retrieving segmentation masks with custom Mask R-CNN, on jetson Xavier AGX

We encounter some difficulties integrating a custom mask r-cnn model into DeepStream 6.0

• Hardware Platform (Jetson / GPU) Jetson Xavier AGX 16Gb
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only) 4.6
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) L4T 32.6.1

• Issue Type( questions, new requirements, bugs)
I built a custom pipeline which takes a h264 video file as input, performs the inference and then records each mask corresponding to the different frames. It works with a custom FCN model, but I want to replace it by a custom Mask R-CNN one. This one has been trained with TAO-Toolkit and then exported to engine format from the custom DeepStream Pipeline. It was trained from a resnet50 backbone. When exporting we observe this :

INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT Input           3x832x1344      
1   OUTPUT kFLOAT generate_detections 10x6            
2   OUTPUT kFLOAT mask_fcn_logits/BiasAdd 10x4x28x28      

I followed this tutorial : MaskRCNN — TAO Toolkit 3.22.05 documentation , having downloaded TensorRT OSS, built the necessary libraries…

With the custom pipeline, I am able to retrieve bbox corresponding to detected objects during inference from GST buffer with a probe function, but don’t succeed to retrieve the corresponding masks using mask_params from NvDsObjectMeta, and obtained this error:

TypeError: Unable to convert function return value to Python type! The signature was
      (self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams

I also tried with pyds.NvDsInferSegmentationMeta.cast and obtained this error:

segmeta = pyds.NvDsInferSegmentationMeta.cast(user_data.user_meta_data)
AttributeError: type object 'pyds.NvDsInferSegmentationMeta' has no attribute 'cast'

To narrow down, I follow this : GitHub - NVIDIA-AI-IOT/deepstream_python_apps: DeepStream SDK Python bindings and sample applications and tried to run the But I get the same error as the last one

Here is the custom python pipeline with use of NvDsObjectMeta mask_params : (5.4 KB)
Here is the nvinfer config file config_v1.txt (1.1 KB)

Can you help us to solve this issue ? Do you have any idea of the source of the problem ?

Looking forward to your feedback

please refer to Accessing Mask Parameters from NvOSD_MaskParams in Python (not using Triton) - #6 by avnguyen213

Thanks for your reply

I have already downloaded and installed deepstream_python_apps. As mentioned in Accessing Mask Parameters from NvOSD_MaskParams in Python (not using Triton), I modified the line

.def_readwrite("mask_params", &NvDsObjectMeta::mask_params)

in bindnvdsmeta.cpp file into :

.def_property_readonly("mask_params", [](NvDsObjectMeta &self) -> py::array {
                   auto dtype = py::dtype(
                   auto base = py::array(dtype,
                   return py::array(dtype, {self.mask_params.size},
                         , base);

But I always get the same error:

TypeError: Unable to convert function return value to Python type! The signature was
      (self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams

I also tried to add in bindnvosd.cpp :

        py::class_<NvOSD_MaskParams>(m, "NvOSD_MaskParams",
                .def_readwrite("data", &NvOSD_MaskParams::data)
                .def_readwrite("size", &NvOSD_MaskParams::size)
                .def_readwrite("threshold", &NvOSD_MaskParams::threshold)
                .def_readwrite("width", &NvOSD_MaskParams::width)
                .def_readwrite("height", &NvOSD_MaskParams::height)
                     [](void *data) {
                         return (NvOSD_MaskParams *) data;

                     [](size_t data) {
                         return (NvOSD_MaskParams *) data;

But this error occurs when running make:

error: 'MaskParamsDoc' is not a member of 'pydsdoc::NvOSD'

Please could you provide us more information on the files to be modified ?


I also modified the pydocumentation.h, adding to the following of “namespace RectParamsDoc {}” :

        namespace MaskParamsDoc
            constexpr const char* descr = R"pyds(
                Holds the mask parameters for the object.

                :ivar data: *float*, Holds mask data for the object.
                :ivar size: *int*, Holds mask size.
                :ivar threshold: *float*, Holds threshold for binarisation.
                :ivar width: *int*, Holds mask width.
                :ivar height: *int*, Holds mask height.)pyds";
            constexpr const char* cast=R"pyds(cast given object/data to :class:`NvOSD_MaskParams`, call pyds.NvOSD_MaskParams.cast(data))pyds";

So the last mentioned error disappear and the first one comes back :

TypeError: Unable to convert function return value to a Python type! The signature was
	(self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams

I enclose the 3 modified files :
pydocumentation.h (126.3 KB)
bindnvosd.cpp (19.9 KB)
bindnvdsmeta.cpp (28.1 KB)

Do you have any idea to solve this issue ?
We look forward to hearing from you

please refer to deepstream_python_apps/bindings at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub, after changed need to rebuild bindings.

Yes, I did.
Do I have to copy the generated library to another directory?

I have the same problem as @sco .To add my experience here:

Modifying the files pydocumentation.h and bindnvosd.cpp I am able to acces to the obj_meta.mask_params. If I print some of its attributes I get:

            type: <class 'pyds.NvOSD_MaskParams'>
            data: -9.71875
            size: 3136
            threshold: 0.0
            width: 28
            height: 28

Probably this values make no sense, since only the ‘data’ value changes in all the masks. I have also tried to get the mask data in the following way:

seg_meta=pyds.NvDsInferSegmentationMeta.cast(obj_meta.mask_params) # object of class pyds.NvDsInferSegmentationMeta

which leads to Segmentation Fault.

Note that, acording to: get_segmentation_masks — Deepstream Deepstream Version: 6.1 documentation, the method get_segmentation_masks expects an object of class pyds.NvDsInferSegmentationMeta (like seg_meta) and returns a numpy array.

What’s wrong here?

BTW, If I print some attributes of seg_meta I get:

            type: <class 'pyds.NvDsInferSegmentationMeta'>
            classes: 470023936
            width: 127
            height: 3136

about bindings, this link helps, Get unique_component_id for segmentation meta - #17 by sandeep.yadav.07780

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.