How to draw masks with python

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

I’m using a Jetson Orinano device with DeepStream 7.0.
When using deepstream-app -c deepstream_app_config.txt to call the yolov8seg model,

Here it is./DeepStream-Yolo-Seg/config_infer_primary_yoloV8_seg.txt:

Here it is./DeepStream-Yolo-Seg/config_infer_primary_yoloV8_seg.txt:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=yolov8s-seg.onnx
model-engine-file=yolov8s-seg.onnx_b1_gpu0_fp32.engine
#int8-calib-file=calib.table
labelfile-path=labels.txt
batch-size=1
network-mode=0
num-detected-classes=80
interval=0
gie-unique-id=1
process-mode=1
network-type=3
cluster-mode=4
maintain-aspect-ratio=1
symmetric-padding=1
#workspace-size=2000
parse-bbox-instance-mask-func-name=NvDsInferParseYoloSeg
custom-lib-path=nvdsinfer_custom_impl_Yolo_seg/libnvdsinfer_custom_impl_Yolo_seg.so
output-instance-mask=1
segmentation-threshold=0.5

[class-attrs-all]
pre-cluster-threshold=0.25
topk=100

This looks great and the mask is translucent and overlays the target area, but I used the following element in python to reproduce this: nvarguscamerasrc-nvvidconv-capsfilter-tee-queue-nvstreammux-nvinfer-nvdsosd-nvjpegenc-appsink (After invoking the CSI camera and inferring in real time to output the MJPEG stream). I added a probe to nvdsosd to find the outline of the mask from the metadata and plot it on the image frame, the function code is as follows. But I don’t think it’s a good way to do it, because I can’t find a function that can directly fill the mask area as a translucent color, so I use the method of drawing the dots myself. It’s slow, very slow, very, very stuttering, and doesn’t work. This is shown in the figure below.
image

def parse_seg_mask_from_meta(frame_meta, obj_meta):
# 获取展平的浮点数置信度掩码
data = obj_meta.mask_params.get_mask_array()

# 获取掩码的高度和宽度
mask_height, mask_width = obj_meta.mask_params.height, obj_meta.mask_params.width

# 将一维数据重塑为 2D 浮点数掩码
confidence_mask = data.reshape((mask_height, mask_width))

# 获取物体边界框参数
bbox_left = obj_meta.rect_params.left
bbox_top = obj_meta.rect_params.top
bbox_width = obj_meta.rect_params.width
bbox_height = obj_meta.rect_params.height

# 计算缩放因子，将掩码大小映射到边界框大小
scale_x = bbox_width / mask_width
scale_y = bbox_height / mask_height

# 设置阈值，将置信度掩码转换为二值掩码
threshold = 0.5
binary_mask = (confidence_mask > threshold).astype(np.uint8)

# 获取显示元数据以在帧上绘制
batch_meta = frame_meta.base_meta.batch_meta
display_meta = pyds.nvds_acquire_display_meta_from_pool(batch_meta)

# 用于在掩码区域进行涂色
color = (0, 255, 0)  # 绿色填充，可以根据需要更改

# 创建一个列表来存储所有要绘制的点
points_to_draw = []

# 遍历掩码并将识别物体区域映射到帧上
for y in range(mask_height):
    for x in range(mask_width):
        if binary_mask[y, x] == 1:  # 如果掩码中该点的值为1，则表示物体区域
            xc_original = int(bbox_left + x * scale_x)
            yc_original = int(bbox_top + y * scale_y)
            points_to_draw.append((xc_original, yc_original))

# 统一更新显示元数据，而不是在每个点处都调用
for point in points_to_draw:
    xc_original, yc_original = point
    if display_meta.num_circles >= MAX_ELEMENTS_IN_DISPLAY_META:
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
        display_meta = pyds.nvds_acquire_display_meta_from_pool(batch_meta)

    # 设置圆形参数，绘制成小圆点，模拟涂色
    circle_params = display_meta.circle_params[display_meta.num_circles]
    circle_params.xc = xc_original
    circle_params.yc = yc_original
    circle_params.radius = 2
    circle_params.circle_color.red = 0.0
    circle_params.circle_color.green = 1.0  # 绿色
    circle_params.circle_color.blue = 1.0
    circle_params.circle_color.alpha = 0.05
    display_meta.num_circles += 1

# 将display_meta添加到帧中
pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)

Later, I learned about the python sample program about deepstream-segmentation in the official sample code, and I learned that it uses nvsegvisual, which is a special element for drawing masks, which is the effect after running through the sample code (unet model), and I noticed that the colors are opaque, and I can’t change the color in this nvsegvisual element. And I changed the infer configuration file in the example code to the yolob8seg model I used above, and the result did not output the mask. So I’m guessing that the deepstream-app -c command to use the yolov8seg model doesn’t use the unit nvsegvisual element? Or how it’s used.

So I’m asking for help here:

  1. What is the effect of deepstream-app -c config.txt this command using the element that appears translucent mask to cover the target area and respond quickly.
    2.How the nvsegvisual component changes the color and translucency of the mask.
  2. Why use deepstream_python_apps/apps/deepstream-segmentation/deepstream_segmentation.py at master · NVIDIA-AI-IOT/deepstream_python_apps (github.com) This example code is changed to the yolov8seg configuration file, and it is displayed without a mask after it is run.
3

Please try to comment in English for the convenience of more people to refer to.

You can refer to our FAQ to get the graph of the pipeline. This graph shows in detail which plugins are used. You can modify our demo based on the graph.

You can set alpha parameter to the nvsegvisual.

4

How to draw mask by using yolov8-seg model in python - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums
Sorry,Sir.This is my question in English.
And I try to set the alpha parameter to the nvsegvisual.

    nvsegvisual_unet = Gst.ElementFactory.make("nvsegvisual", "nvsegvisual_unet")
    if not nvsegvisual_unet:
        sys.stderr.write("Unable to create nvsegvisual_unet\n")
    nvsegvisual_unet.set_property('batch-size', 1)
    nvsegvisual_unet.set_property('width', 512) 
    nvsegvisual_unet.set_property('height', 512)
    nvsegvisual_unet.set_property('alpha', 0.5)

But setting alpha in this way doesn’t work. I checked the document “Gst-nvsegvisual — DeepStream documentation 6.4 documentation”, and there is no mention of transparency properties.

5

And thank you for your reply for my third question,I succeed in getting the graph of the pipline.~~~ TvT

6

I have obtained the table of pipeline elements, but I still can’t figure out how the mask area color is drawn. How is this done?

7

The color is drawn by the nvsegvisual plugin. Currently, it cannot be changed by set parameters outside. We can only support the transparency parameters now.
g_object_set (G_OBJECT (segvisual), “alpha”, alpha, NULL);.

8

Thanks for your reply.I try to set’ alpha’ in python ,but isn’t useful.
nvsegvisual_unet.set_property(‘alpha’, 0.5)
That is why?

9

This paramter refers to the transparency of the mask relative to the original image. It will only take effect if you set the nvsegvisual.set_property('original-background', 1) parameter.

10

Thank you for your answer. We have a holiday these days. I’ll try it after work. How can I know which attributes of this element can be adjusted? is there any documentation? The documents I found about nvsegvisual don’t have these.

11

You can run the command below directly to check the attributes of this plugin.

gst-inspect-1.0 nvsegvisual