Drawing shapes on top of video output

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi,
I am trying to draw some shapes on top of the video output from sample_test1 source code.
I thought of using cairo overlay[url]https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-cairooverlay.html[/url] Gstreamer plugin to do this.

Following the example provided here[url]https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/cairo/cairo_overlay.c[/url], I tried adding the overlay (and the adaptors) between the nvosd and the sink.
However after adding the plugin, the video doesn’t come up and after running the for a couple of seconds the program stops.

What am I doing wrong here? Are there any other options to do this?
Cheers!

2

Hi Marshall,
The Data between different component is Nv private structure(NvBuferSurface), which wrap the GPU buffer’s pointer.

Buffer between different component is GPU buffer, make sure your library is GPU buffer based.

Are you wanting to draw some special shape in the probe function of nvosd’s src sink?
Could you give a detail about what you want to do, then I can give you some advice about how to do.

Thanks
wayne zhu

3

Hi Wayne,
I want to draw a polygonal box (no need to be filled with color) on top of the video to show the region of interest.
Just like how nvosd can draw bounding boxes of the object, I thought it might be possible to pass the ROI data through a new NvDsMeta structure into nvosd. I failed while attempting this which lead me trying a different method by using the Gstreamer plug-in.

I am not sure about the cairo overlay plug-in buffer type

Thanks,
Marshall

4

Hi Marshalltata,

For this case, you need add a probe function src/sink’s pad.

In probe function, get gstbuffer → get NvbufSurface → Get GPU address → copy to CPU buffer → call cairo → copy to GPU buffer.

You can refer gst-dsexample for how to use NvbufSurface

Thanks
wayne zhu

5

Hi Waynezhu,

Thanks for the idea!
I’ll try experimenting by myself first and I will post again any progress or more questions at later time.

Thanks,
Marshall

6

Hey Marshalltata,

were you able to get it to work? I’m also trying to draw shapes and other textures on top of a video output and am having a bit of trouble figuring out where to start.

-Mercy

7

Hey Marshalltata,

I am trying to create a ROI to filter the unwanted detection and I face similar problem. Would you mind to share how you do it?

Thank you.

8

Hey,

sorry for the late reply, I still haven’t managed to make it work while trying it. And since then Deepstream 3.0 was released and it seems that it has the ability to draw areas on screen using GPS positioning. You might want to see if it satisfies your project’s needs.

Regards,
Marshall

9

Hi Marshalltata,

Deepstream 3.0 shows an example of drawing ROI for smart parking. But this solution is look like specific to smart parking use case with GPS positioning and the information of line input is not mentioned in the example. Is there any simple method to draw a line or ROI on top video?

Regards,
Nero

10

Hi Nero,

you might want to look at the gstreamer plug-in I mentioned on the first post. Waynezhu had kindly explained what needs to be done to make it work together with Deepstream, which sadly, I still haven’t managed to get it work.

Please keep in mind this is just a suggestion and there might be better alternatives.

Regards,
Marshall

11

2 methods for this work:

  1. add a probe on src/sink pad of exisiting plugins and extract the NvBuffSurface to get raw frame data, and draw your shapes with lib like opencv;

  2. write a custom plugin (derived from GstBaseTransform) and extract the NvBuffSurface structure in transform_ip function, also get the raw frame data, draw your shapes with lib like opencv.

the source code in gstdsexample(DeepStream 3.0 Release folder) has show you how to convert raw frame data into cv::Mat :

// Process to get the output
output = DsExampleProcess (dsexample->dsexamplelib_ctx,dsexample->cvmat->data);

you can pass the cv::Mat into a custom library from your custom plugin. it is very easy.

12

Thank you Zhouzhi9 for the possible alternative solutions for drawing shapes on top of video output

13

I found this thread very helpful in getting started on how to draw shapes on top of video output using opencv.

https://devtalk.nvidia.com/default/topic/1045453/deepstream-sdk-on-jetson/how-to-draw-lanes-on-the-video/post/5315160/?offset=15#5317765