Draw masks using nvdsosd and python bindings

• Jetson Orin NX
• Deepstream 6.3
• JetPack 5.1.2

Hello!

I am creating a custom plugin (written in python) to get familiar with writing to the deepstream metadata and then use the nvdsosd plugin to draw boxes/text/masks.

So far, text and boxes work fine, but I cannot get the instance masks working. Here’s a snippet of my code

class GstCustomDraw(GstBase.BaseTransform):
    # omitted code

    def load_mask(self):
        """ Load and preprocess the black & white mask image """
        if not self.mask_path:
            print("No mask path provided.")
            return
        
        try:
            print(f"Loading mask from: {self.mask_path}")
            img = cv2.imread(self.mask_path, cv2.IMREAD_GRAYSCALE)
            if img is None:
                print("Failed to load mask image!")
                return
            
            _, self.mask = cv2.threshold(img, 128, 255, cv2.THRESH_BINARY)
            self.mask = np.uint8(self.mask)
            print("Mask successfully loaded!")
        except Exception as e:
            print(f"Error loading mask: {e}")

     def add_object_meta(self, batch_meta, frame_meta):
        """ Add a square object to the metadata """
        obj_meta = pyds.nvds_acquire_obj_meta_from_pool(batch_meta)
        
        # Set object metadata
        obj_meta.confidence = 0.85  # Hardcoded confidence
        obj_meta.class_id = 1  # Assign a class ID
        obj_meta.obj_label = "Custom Object"
        
        # Set rect params
        rect_params = obj_meta.rect_params
        rect_params.left = frame_meta.source_frame_width // 4
        rect_params.top = frame_meta.source_frame_height // 4
        rect_params.width = self.square_size
        rect_params.height = self.square_size
        rect_params.border_width = 3
        rect_params.has_bg_color = 1
        rect_params.bg_color.set(*self.square_color)  # Use configured color
        rect_params.border_color.set(0.0, 1.0, 0.0, 1.0)  # Green border

        # Assign mask data
        mask_params = obj_meta.mask_params
        mask_params.threshold = 128
        mask_params.width = self.mask.shape[1]
        mask_params.height = self.mask.shape[0]
        mask_params.size = self.mask.size
        mask_params.data[:] = self.mask.flatten()

        # set text params
        text_params = obj_meta.text_params
        text_params.display_text = "Custom Object"
        text_params.x_offset = int(rect_params.left)
        text_params.y_offset = int(rect_params.top)
        text_params.font_params.font_name = "Serif"
        text_params.font_params.font_size = 15
        text_params.font_params.font_color.set(0.0, 0.0, 0.0, 1.0)  # Black
        text_params.set_bg_clr = 1
        text_params.text_bg_clr.set(1.0, 0.0, 0.0, 1.0)  # Red background

        pyds.nvds_add_obj_meta_to_frame(frame_meta, obj_meta, None)

    # omitted code

I have found examples on how to read from obj_meta.mask_params, but nothing on how to write to the NvDsMeta.

Am I doing something wrong?

Here is the gstreamer pipeline I am using to test

gst-launch-1.0 videotestsrc pattern=ball animation-mode=1 ! \
"video/x-raw, width=1280, height=960, format=RGBA" ! \
nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! \
nvstreammux0.sink_0 nvstreammux name=nvstreammux0 batch-size=1 width=1280 height=960 live-source=TRUE ! \
queue ! customdraw square-size=200 mask-path=mask.png ! \
queue ! nvdsosd display-clock=1 display-bbox=1 display-mask=1 ! \
queue ! nvvideoconvert ! <some_sink>

This is a screenshot of what I am getting (the display_meta code is ommited from the snipped above).

Thank you!

Please refer to this code snippet,mask_params is usually filled by nvinfer. If you want to modify or add in python, you need to use alloc_mask_array/get_mask_array

while l_obj is not None:
            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break
            obj_counter[obj_meta.class_id] += 1
            obj_meta.rect_params.border_color.set(0.0, 0.0, 1.0, 0.8) #0.8 is alpha (opacity)

            # fill mask
            obj_meta.mask_params.width = 180
            obj_meta.mask_params.height = 180
            obj_meta.mask_params.threshold = 0.2
            mask = obj_meta.mask_params.alloc_mask_array()
            mask.fill(0.5)
            try: 
                l_obj=l_obj.next
            except StopIteration:
                break

https://docs.nvidia.com/metropolis/deepstream/dev-guide/python-api/PYTHON_API/NvOSD/NvOSD_MaskParams.html

Hi @junshengy,

I tried it, but I got an error because I am using DeepStream 6.3 instead of 7.1 NvOSD_MaskParams — Deepstream Deepstream Version: 6.3 documentation

AttributeError: 'pyds.NvOSD_MaskParams' object has no attribute 'alloc_mask_array'
Traceback (most recent call last):
  File "/home/power/deepstream-tests/python/customdraw.py", line 181, in do_transform_ip
    self.add_object_meta(batch_meta, frame_meta)
  File "/home/power/deepstream-tests/python/customdraw.py", line 122, in add_object_meta
    mask = mask_params.alloc_mask_array()

Perhaps there is another way to do this in DS 6.3?

Thanks!
Kevin

If you must use DS-6.3, you can consider porting the above pyds binding from DS-7.1 to DS-6.3 , and then recompile and install

I have tested the above code on DS-7.1 and it works.

Dear @junshengy

Thanks for the info. If not possible with the mask_params, is there another plugin which can help? For example this one? Gst-nvsegvisual — DeepStream documentation

Do you happen to have info/references on how to write (using python) to the NvDsInferSegmentationMeta?

KS

What is your goal? nvsegvisual is used for semantic segmentation and is responsible for rendering NvDsInferSegmentationMeta to a video mask.

And obj_meta.mask_params is used for instance segmentation, and they do completely different things.

This is an example of semantic segmentation. The mask data is output by the model, not generated in the python program

/opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-segmentation/deepstream_segmentation.py

Hi @junshengy

In my use case, I need to apply a mask to filter out detections in certain areas and visualize it for debugging/testing purposes.

I’d like to handle everything within a single plugin, which is why I was aiming to use nvdsosd. Currently, I’m trying to read a black-and-white image and display it somehow using the python bindings.

This deepstream_python_apps/apps/deepstream-segmentation/deepstream_segmentation.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub is great for reading from NvDsInferSegmentationMeta, but I cannot find how to write to it. Would be great if you could point me to a simple example or snippet.

Best,
Kevin

NvDsInferSegmentationMeta is useless for your goal, it can only be used with nvsegvisual to show the categories in the image as different colors. just like

If you port the above code correctly, you can achieve the following effect

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.