Hi, We encounter some difficulties integrating a custom mask r-cnn model into DeepStream 6.0 • Hardware Platform (Jetson / GPU) Jetson Xavier AGX 16Gb • DeepStream Version 6.0 • JetPack Version (valid for Jetson only) 4.6 • TensorRT Version 8.0.1.6 • NVIDIA GPU Driver Version (valid for GPU only) L4T 32.6.1 • Issue Type( questions, new requirements, bugs) I built a custom pipeline which takes a h264 video file as input, performs the inference and then records each mask corresponding to the different frames. It works with a custom FCN model, but I want to replace it by a custom Mask R-CNN one. This one has been trained with TAO-Toolkit and then exported to engine format from the custom DeepStream Pipeline. It was trained from a resnet50 backbone. When exporting we observe this : INFO: [Implicit Engine Info]: layers num: 3 0 INPUT kFLOAT Input 3x832x1344 1 OUTPUT kFLOAT generate_detections 10x6 2 OUTPUT kFLOAT mask_fcn_logits/BiasAdd 10x4x28x28 I followed this tutorial : MaskRCNN — TAO Toolkit 3.22.05 documentation , having downloaded TensorRT OSS, built the necessary libraries… With the custom pipeline, I am able to retrieve bbox corresponding to detected objects during inference from GST buffer with a probe function, but don’t succeed to retrieve the corresponding masks using mask_params from NvDsObjectMeta, and obtained this error: TypeError: Unable to convert function return value to Python type! The signature was (self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams I also tried with pyds.NvDsInferSegmentationMeta.cast and obtained this error: segmeta = pyds.NvDsInferSegmentationMeta.cast(user_data.user_meta_data) AttributeError: type object 'pyds.NvDsInferSegmentationMeta' has no attribute 'cast' To narrow down, I follow this : GitHub - NVIDIA-AI-IOT/deepstream_python_apps: DeepStream SDK Python bindings and sample applications and tried to run the deepstream-segmentation.py. But I get the same error as the last one Here is the custom python pipeline with use of NvDsObjectMeta mask_params : custom_pipeline.py (5.4 KB) Here is the nvinfer config file config_v1.txt (1.1 KB) Can you help us to solve this issue ? Do you have any idea of the source of the problem ? Looking forward to your feedback

please refer to deepstream_python_apps/bindings at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub, after changed need to rebuild bindings.

Thanks for your reply I have already downloaded and installed deepstream_python_apps. As mentioned in Accessing Mask Parameters from NvOSD_MaskParams in Python (not using Triton), I modified the line .def_readwrite("mask_params", &NvDsObjectMeta::mask_params) in bindnvdsmeta.cpp file into : .def_property_readonly("mask_params", [](NvDsObjectMeta &self) -> py::array { auto dtype = py::dtype( py::format_descriptor ::format()); auto base = py::array(dtype, {self.mask_params.size}, {sizeof(float)}); return py::array(dtype, {self.mask_params.size}, {sizeof(float)}, self.mask_params.data, base); }) But I always get the same error: TypeError: Unable to convert function return value to Python type! The signature was (self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams I also tried to add in bindnvosd.cpp : py::class_ (m, "NvOSD_MaskParams", pydsdoc::NvOSD::MaskParamsDoc::descr) .def(py::init<>()) .def_readwrite("data", &NvOSD_MaskParams::data) .def_readwrite("size", &NvOSD_MaskParams::size) .def_readwrite("threshold", &NvOSD_MaskParams::threshold) .def_readwrite("width", &NvOSD_MaskParams::width) .def_readwrite("height", &NvOSD_MaskParams::height) .def("cast", [](void *data) { return (NvOSD_MaskParams *) data; }, py::return_value_policy::reference, pydsdoc::NvOSD::RectParamsDoc::cast) .def("cast", [](size_t data) { return (NvOSD_MaskParams *) data; }, py::return_value_policy::reference, pydsdoc::NvOSD::MaskParamsDoc::cast); But this error occurs when running make: error: 'MaskParamsDoc' is not a member of 'pydsdoc::NvOSD' pydsdoc::NvOSD::MaskParamsDoc::descr) Please could you provide us more information on the files to be modified ?

Hi, I also modified the pydocumentation.h, adding to the following of “namespace RectParamsDoc {}” : namespace MaskParamsDoc { constexpr const char* descr = R"pyds( Holds the mask parameters for the object. :ivar data: *float*, Holds mask data for the object. :ivar size: *int*, Holds mask size. :ivar threshold: *float*, Holds threshold for binarisation. :ivar width: *int*, Holds mask width. :ivar height: *int*, Holds mask height.)pyds"; constexpr const char* cast=R"pyds(cast given object/data to :class:`NvOSD_MaskParams`, call pyds.NvOSD_MaskParams.cast(data))pyds"; } So the last mentioned error disappear and the first one comes back : TypeError: Unable to convert function return value to a Python type! The signature was (self: pyds.NvDsObjectMeta) -> _NvOSD_MaskParams I enclose the 3 modified files : pydocumentation.h (126.3 KB) bindnvosd.cpp (19.9 KB) bindnvdsmeta.cpp (28.1 KB) Do you have any idea to solve this issue ? We look forward to hearing from you

Yes, I did. Do I have to copy the generated pyds.so library to another directory?

I have the same problem as @sco .To add my experience here: Modifying the files pydocumentation.h and bindnvosd.cpp I am able to acces to the obj_meta.mask_params. If I print some of its attributes I get: mask_params: type: data: -9.71875 size: 3136 threshold: 0.0 width: 28 height: 28 Probably this values make no sense, since only the ‘data’ value changes in all the masks. I have also tried to get the mask data in the following way: seg_meta=pyds.NvDsInferSegmentationMeta.cast(obj_meta.mask_params) # object of class pyds.NvDsInferSegmentationMeta seg=pyds.get_segmentation_masks(NvDsInferSegmentationMeta) which leads to Segmentation Fault. Note that, acording to: gst_element_send_nvevent_new_stream_reset — Deepstream Deepstream Version: 6.1.1 documentation, the method get_segmentation_masks expects an object of class pyds.NvDsInferSegmentationMeta (like seg_meta) and returns a numpy array. What’s wrong here? BTW, If I print some attributes of seg_meta I get: seg_meta: type: classes: 470023936 width: 127 height: 3136

about bindings, this link helps, Get unique_component_id for segmentation meta - #17 by sandeep.yadav.07780

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks

Error when retrieving segmentation masks with custom Mask R-CNN, on jetson Xavier AGX

Accelerated Computing Intelligent Video Analytics DeepStream SDK

fanzh July 20, 2022, 10:08am 3

please refer to Accessing Mask Parameters from NvOSD_MaskParams in Python (not using Triton) - #6 by avnguyen213

Topic		Replies	Views
NvOSD_MaskParams Missing Documentation DeepStream SDK	1	1010	August 22, 2022
Failed to accessing segmentation data for PeopleSegNet(mrcnn) TAO Toolkit	4	631	July 6, 2022
Accessing segmentation metadata for Maskrcnn DeepStream SDK	13	1158	August 25, 2021
NvOSD_MaskParams - missing from Python Bindings DeepStream SDK	2	468	August 1, 2022
Unable to Get Segmentation Masks Values and Colors DeepStream SDK	7	641	February 20, 2024
Unable to visualize and extract masks when using Mask-RCNN with default Deepstream App DeepStream SDK	8	937	April 17, 2020
Get TLT Mask-RCNN output masks into python / numpy DeepStream SDK	1	528	March 20, 2021
Access metadata for segmentation mask DeepStream SDK nvbugs , python	3	1042	August 17, 2020
Getting Segmentation Meta Data (NvDsInferSegmentationMeta) of custom segmentation model DeepStream SDK	4	1158	November 16, 2020
Modify NvDsInferObjectDetectionInfo for get mask from Mask RCNN DeepStream SDK	2	539	May 25, 2020

Error when retrieving segmentation masks with custom Mask R-CNN, on jetson Xavier AGX

Related topics