求助用python编写Gstreamer调用yolov8-seg模型如何绘制掩码

我在使用jetson orinano设备 deepstream7.0。
使用 deepstream-app -c deepstream_app_config.txt 来调用yolov8seg模型时,

下面是./DeepStream-Yolo-Seg/config_infer_primary_yoloV8_seg.txt:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=yolov8s-seg.onnx
model-engine-file=yolov8s-seg.onnx_b1_gpu0_fp32.engine
#int8-calib-file=calib.table
labelfile-path=labels.txt
batch-size=1
network-mode=0
num-detected-classes=80
interval=0
gie-unique-id=1
process-mode=1
network-type=3
cluster-mode=4
maintain-aspect-ratio=1
symmetric-padding=1
#workspace-size=2000
parse-bbox-instance-mask-func-name=NvDsInferParseYoloSeg
custom-lib-path=nvdsinfer_custom_impl_Yolo_seg/libnvdsinfer_custom_impl_Yolo_seg.so
output-instance-mask=1
segmentation-threshold=0.5

[class-attrs-all]
pre-cluster-threshold=0.25
topk=100

这个显示效果很好掩码是半透明的 而且覆盖在目标区域上,但我用python 使用了以下元素想要复现这个效果:nvarguscamerasrc-nvvidconv-capsfilter-tee-queue-nvstreammux-nvinfer-nvdsosd-nvjpegenc-appsink (调用csi摄像头并实时推理后以输出mjpeg流)。我在nvdsosd加了一个探针,从元数据里找出掩码的轮廓并绘制在图像帧上,函数代码如下。但我认为这不是一个很好的方法,因为我找不到能直接填充掩码区域为半透明颜色的函数,所以我采用自己画点绘制的方法。这很慢非常慢非常非常卡顿,无法正常工作。如下图所示。
image

def parse_seg_mask_from_meta(frame_meta, obj_meta):
# 获取展平的浮点数置信度掩码
data = obj_meta.mask_params.get_mask_array()

# 获取掩码的高度和宽度
mask_height, mask_width = obj_meta.mask_params.height, obj_meta.mask_params.width

# 将一维数据重塑为 2D 浮点数掩码
confidence_mask = data.reshape((mask_height, mask_width))

# 获取物体边界框参数
bbox_left = obj_meta.rect_params.left
bbox_top = obj_meta.rect_params.top
bbox_width = obj_meta.rect_params.width
bbox_height = obj_meta.rect_params.height

# 计算缩放因子,将掩码大小映射到边界框大小
scale_x = bbox_width / mask_width
scale_y = bbox_height / mask_height

# 设置阈值,将置信度掩码转换为二值掩码
threshold = 0.5
binary_mask = (confidence_mask > threshold).astype(np.uint8)

# 获取显示元数据以在帧上绘制
batch_meta = frame_meta.base_meta.batch_meta
display_meta = pyds.nvds_acquire_display_meta_from_pool(batch_meta)

# 用于在掩码区域进行涂色
color = (0, 255, 0)  # 绿色填充,可以根据需要更改

# 创建一个列表来存储所有要绘制的点
points_to_draw = []

# 遍历掩码并将识别物体区域映射到帧上
for y in range(mask_height):
    for x in range(mask_width):
        if binary_mask[y, x] == 1:  # 如果掩码中该点的值为1,则表示物体区域
            xc_original = int(bbox_left + x * scale_x)
            yc_original = int(bbox_top + y * scale_y)
            points_to_draw.append((xc_original, yc_original))

# 统一更新显示元数据,而不是在每个点处都调用
for point in points_to_draw:
    xc_original, yc_original = point
    if display_meta.num_circles >= MAX_ELEMENTS_IN_DISPLAY_META:
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
        display_meta = pyds.nvds_acquire_display_meta_from_pool(batch_meta)

    # 设置圆形参数,绘制成小圆点,模拟涂色
    circle_params = display_meta.circle_params[display_meta.num_circles]
    circle_params.xc = xc_original
    circle_params.yc = yc_original
    circle_params.radius = 2
    circle_params.circle_color.red = 0.0
    circle_params.circle_color.green = 1.0  # 绿色
    circle_params.circle_color.blue = 1.0
    circle_params.circle_color.alpha = 0.05
    display_meta.num_circles += 1

# 将display_meta添加到帧中
pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)

后来我了解官方示例代码里关于deepstream-segmentation的python示例程序,我了解到它使用了nvsegvisual这个专门用来画掩码的元素,这是跑通示例代码(unet模型)后呈现的效果,我注意到颜色都是不透明的,而且我没法在这个nvsegvisual元素里改颜色。并且我将示例代码里的infer配置文件改成上面我使用的yolob8seg模型 ,跑出来的结果未输出掩码。所以我猜测deepstream-app -c 这个命令来使用yolov8seg模型也不是使用单位nvsegvisual这一元素?或者它是怎么使用的。

所以我在此寻求帮助:
1.请问deepstream-app -c config.txt 这一命令是使用了什么元素出现的半透明掩码覆盖在目标区域 并且响应速度很快的效果。
2.nvsegvisual组件如何更改掩码的颜色和半透明程度。
3.为什么使用 deepstream_python_apps/apps/deepstream-segmentation/deepstream_segmentation.py at master · NVIDIA-AI-IOT/deepstream_python_apps (github.com) 这个示例代码 改成yolov8seg的配置文件,跑出来后无掩码呈现。

Please try to comment in English for the convenience of more people to refer to.

You can refer to our FAQ to get the graph of the pipeline. This graph shows in detail which plugins are used. You can modify our demo based on the graph.

You can set alpha parameter to the nvsegvisual.

How to draw mask by using yolov8-seg model in python - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums
Sorry,Sir.This is my question in English.
And I try to set the alpha parameter to the nvsegvisual.

    nvsegvisual_unet = Gst.ElementFactory.make("nvsegvisual", "nvsegvisual_unet")
    if not nvsegvisual_unet:
        sys.stderr.write("Unable to create nvsegvisual_unet\n")
    nvsegvisual_unet.set_property('batch-size', 1)
    nvsegvisual_unet.set_property('width', 512) 
    nvsegvisual_unet.set_property('height', 512)
    nvsegvisual_unet.set_property('alpha', 0.5)  

But setting alpha in this way doesn’t work. I checked the document “Gst-nvsegvisual — DeepStream documentation 6.4 documentation”, and there is no mention of transparency properties.

And thank you for your reply for my third question,I succeed in getting the graph of the pipline.~~~ TvT

I have obtained the table of pipeline elements, but I still can’t figure out how the mask area color is drawn. How is this done?

The color is drawn by the nvsegvisual plugin. Currently, it cannot be changed by set parameters outside. We can only support the transparency parameters now.
g_object_set (G_OBJECT (segvisual), “alpha”, alpha, NULL);.

Thanks for your reply.I try to set’ alpha’ in python ,but isn’t useful.
nvsegvisual_unet.set_property(‘alpha’, 0.5)
That is why?

This paramter refers to the transparency of the mask relative to the original image. It will only take effect if you set the nvsegvisual.set_property('original-background', 1) parameter.

Thank you for your answer. We have a holiday these days. I’ll try it after work. How can I know which attributes of this element can be adjusted? is there any documentation? The documents I found about nvsegvisual don’t have these.

You can run the command below directly to check the attributes of this plugin.

gst-inspect-1.0 nvsegvisual