Gstreamer - muxing metadata over UDP Stream


I am working on a project that would send metadata from my drone together with my video.
I have a CSI camera connected to JetsonNX, and I was using OpenCV with Gstreamer pipeline until now, based on this post:

But I want to mux tracking box data into metadata so I can draw it later on the ground. I was searching around and found out that, for that .ts format is used. However, I failed to find any working example of someone using it using Gstreamer on my Ubuntu PC, not even mentioning JetsonNX.

I checked and GStreamer have some elements for doing that:

gst-inspect-1.0 | grep mpegtsmpegtsdemux:  tsparse: MPEG transport stream parser
mpegtsdemux:  tsdemux: MPEG transport stream demuxer
mpegtsmux:  mpegtsmux: MPEG Transport Stream Muxer
typefindfunctions: video/mpegts: ts, mts
libav:  avmux_mpegts: libav MPEG-TS (MPEG-2 Transport Stream) muxer (not recommended, use mpegtsmux instead)

It would be perfect if I could use it in OpenCV like in that post, as I want to add data that’s in my code.

Can someone help me achieve that?

We don’t have experience in the use-case. There are some idea but not sure if it can work or not. May need to give it a try. Also would see if other users can share experience.

First, in mpegtsmux we can see:

        parsed: true

Looks like the two tracks can be put for additional information. You may search online and see if there is examples about them.

The second possible solution is to put SEI into H264 stream. gst-v4l2 is open source and you can customize nvv4l2h264enc plugin to add the function. To add the information to SEI and append to compressed stream.

I hope you can try your idea, and it will work :)
KLV seems easier, and I found more info about it online. I don’t quite understand SEI.

I looked online but I didn’t find any info on how to achieve that :c

Method 1 may not work in OpenCV, because it requires two appsrc. One is to feed frame data, and the other is to feed metadata. OpenCV supports single appsrc in each cv2.VideoWriter().

Method 2 should work, do you know how to feed SEI NALs? You can implement the code to feed SEI before IDR frames.

Method 1:
If it’s not possible in OpenCV, can I use Gstreamer API for this? And collect the frame into it from OpenCV cv::Mat?

I don’t have any experience with GStreamer API but I’m willing to try :)

Method 2:
Unfortunately, I don’t. Do you have any useful examples?

Method 2 looks more feasible. There are discussions about adding SEI in
h.264 - h264: Add user unregistered SEI message - Stack Overflow

Please take a look and try to apply it to gst-v4l2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.