Get TLT Mask-RCNN output masks into python / numpy

[daniel80]

• Hardware Platform: GPU
• DeepStream Version: 5.1
• TensorRT Version: 7.2.1
• NVIDIA GPU Driver Version (valid for GPU only): 460.56
• Issue Type: question

Is it possible to access the MaskRCNN output masks from python?

I am using the MaskRCNN of the Transfer Learning Toolkit (MaskRCNN — Transfer Learning Toolkit 3.0 documentation), and created a pipeline in python that closely follows the deepstream_python_apps (GitHub - NVIDIA-AI-IOT/deepstream_python_apps: A project demonstrating use of Python for DeepStream sample apps given as a part of SDK (that are currently in C,C++).).
In the config file I set output-instance-mask=1 but I don’t know how to get this output from within python (or if it is possible at all).
I think Mask RCNN has a different metadata output (NvDsInferInstanceMaskInfo) compared to the deepstream_python_apps/apps/deepstream-segmentation sample app, but can’t figure out how I can access this from within python.

[mchi]

Hi @daniel80 ,
Could reer to DS python sample - deepstream_python_apps/apps/deepstream-ssd-parser at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

1. Register a pgie srcpad probe function
   pgiesrcpad.add_probe(Gst.PadProbeType.BUFFER, pgie_src_pad_buffer_probe, 0)

2. In srcpad probe function, call nvds_infer_parse_custom_tf_ssd(layers_info, detection_params, box_size_param, nms_param), here layers_info is NvDsInferLayerInfo , in which property buffer is Pointer to the buffer for the layer data. So, the data pointed by buffer is the raw output data of the network.

3. have tensor outputs as meta in GstBuffer, you need to enable “output-tensor-meta=1”.
   Doc - https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.html#wwpID0E0YFB0HA , in DS python, it’s deepstream_python_apps/dstest_ssd_nopostprocess.txt at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub