How do I draw a more complicated overlay on a deepstream video?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Please provide complete information as applicable to your setup.

• Hardware Platform: NVIDIA L40 GPU
• DeepStream 7.0
• NVIDIA GPU Driver Version 535.183.06

Hi, my team are trying to expand on the NvOSD overlay in the Python DeepStream API, such as rendering icons and more complicated shapes in the form of a UI directly within the deepstream pipeline, but we’re struggling to get anything to display beyond the basic bounding box and text parameters of the NvOSD. This is heavily based on the DeepStream Python Apps examples.

When creating a custom probe on an overlay element, and trying to modify the NvBufSurface via pyds.get_nvds_buf_surface, if the NvBufSurface is modified in any way then the program cleanly crashes. This happens whether using opencv to draw a line on the image, or even just raw numpy to, for example make half of the image red (as an example):

def tiler_sink_pad_buffer_probe(pad, info, u_data):
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))

    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break

        l_obj = frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break
            n_frame = pyds.get_nvds_buf_surface(
                hash(gst_buffer), frame_meta.batch_id
            )

            height, width, channels = n_frame.shape
            # Set the first half of the image to red
            n_frame[:, : width // 2, 0] = 255  # Set the red channel to 255
            n_frame[:, : width // 2, 1] = 0  # Set the green channel to 0
            n_frame[:, : width // 2, 2] = 0  # Set the blue channel to 0
            n_frame[:, : width // 2, 3] = 128  # Set the alpha channel to 128

Which is connected with the following links in the pipeline:

            streammux.link(pgie)
            pgie.link(tracker)
            tracker.link(analytics)
            analytics.link(demux)
            demux.link(nvvidconv)
            nvvidconv.link(filter1)
            filter1.link(tiler)
            tiler.link(nvvidconv1)
            nvvidconv1.link(nvosd)
            nvosd.link(tee)

            tiler_sink_pad = tiler.get_static_pad("sink")
            tiler_sink_pad.add_probe(
                Gst.PadProbeType.BUFFER, tiler_sink_pad_buffer_probe, 0
            )

How do we modify the buffer surface to draw our own custom bounding boxes with icons and custom shapes onto the frames of the video stream? If I remove the code where I modify the pixels of n_frame, then the video displays as expected with the default OSD, but if I add it then I get a black screen and the program halts. The same result occurs if I use anything else to modify n_frame in any manner.

I know that I can use a probe to modify the NvOSD element’s display meta, but I need a solution to modify the frame pixel data in a more custom/direct manner to allow for adding icons and more complicated shapes (like rounded squares or triangles).

Any advice on how to fix the above code, or even an alternate approach with a similar result would be greatly appreciated.

3

Could you refer to our deepstream_imagedata-multistream.py?

4

As mentioned, the code I have is heavily based upon this example. But I cannot seem to modify the data from NvBufSurface. What could cause crashes when trying to modify this frame data?
Is modifying the buffer surface manually like this even the best approach for the more custom overlay that I’m attempting?

5

It may be that the image data is not in the right format. Depending on your changes, you need to make sure the format is RGBA.

No. We still recommend that you use our display meta. Like if you need to draw polygons, draw them yourself using lines.

6

Yes I can confirm that the image data is in the format RGBA - I can see that n_frame correctly has 4 channels.

For drawing polygons directly on the display_meta, I can see the docs for NvDsDisplayMeta. With this I can draw lines and circles like so

        display_meta.num_lines = 1
        line_param = display_meta.line_params[0]
        line_param.x1 = 100
        line_param.x2 = 200
        line_param.y1 = 100
        line_param.y2 = 200
        line_param.line_width = 10
        line_param.line_color.set(1.0, 1.0, 1.0, 1.0)

        display_meta.num_circles = 1
        circle_param = display_meta.circle_params[0]
        circle_param.xc = 300
        circle_param.yc = 300
        circle_param.radius = 50
        circle_param.circle_color.set(1.0, 1.0, 1.0, 1.0)
        circle_param.has_bg_color = 1
        circle_param.bg_color.set(1.0, 0.0, 0.0, 1.0)

But the display_meta is for the whole frame. How do I create custom shapes / icons per object in the frame? From the NvDsFrameMeta I can access the NvDsObjectMeta per object in the frame, and then on top of this I have the standard bounding box, but this can only modify the single box and text. How do I draw more of a custom UI per object? Is the best practice here to get the location of the rectangular bounding box in the object_meta, and then for each object draw a shape at that location in the display_meta?

My goal is to draw shapes like triangles above each object in the frame, and see if I can get icons drawn above that.

7

Yes. If you want to draw other shapes, you need to confirm the coordinates of each object first.

If you want to draw that yourself, we still suggest that you first run our deepstream_imagedata-multistream.py sample first to learn how to draw that with OpenCV.

8

Yes, as mentioned above my code is heavily based upon this example - my problem being that modifying the n_frame from pyds.get_nvds_buf_surface causes a crash with no output or error. The data can be read fine it seems, but any attempt to modify it, either with OpenCV or even just manually through numpy, causes a crash. I don’t know if my link order in the pipeline could be the cause here.

So is the best approach to manually draw each shape and line using the display_meta, as mentioned above, or is it to use OpenCV to manipulate the pixel data in the n_frame as you say now?

9

If you just want to draw the triangle or other polygons with lines, we suggest you use display_meta. You need to confirm the starting and ending coordinates of each line by yourself, and then draw it directly with our display_meta.

10

Thanks for the help here. That should be straightforward for simple lines and shapes.

What is the best way to get icons into the OSD? For example transposing an svg/png onto a given frame of the stream?

11

We do not currently support a similar blending function.

12

I’m wondering in the meantime if something like cv::cuda::alphaComp or some Alpha blending kernel within the spot where opencv is being called would work.

13

We don’t have a similar sample at present, you can have a try.

14

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks