How to get the SEI payload in Deepstream pipeline

[wei.zhang12]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson)
• DeepStream Version 6.3
• JetPack Version (5.1.2)
• TensorRT Version 8.5.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

From this topic, How to get rtsp sei payload?
It seems the dGPU platform can extract SEI by nvv4l2decoder .
But I am using the Jetson device, I want to know how to do?

[Fiona.Chen]

There is no such function in Jetson.

You may refer to GStreamer sources. GstH264Parser (gstreamer.freedesktop.org)

[wei.zhang12]

Do you mean I shall add some customized implementation inside GstH264Parser? And re-compile and install it on Jetson device? Do you have a guide, how to do this?

[wei.zhang12]

BTW, I am using nvurisrcbin to connect with multi streams. And I don’t know how do I connect this nvurisrcbin with h264parser.

[Fiona.Chen]

If you are using nvurisrcbin, the “extract-sei-type5-data” property is supported.

[wei.zhang12]

I refer to deepstream-test3 python app and I add extract-sei-type5-data inside create_source_bin. And then I do I get this SEI payload?

And I get below error GstURIDecodeBin' does not have property extract-sei-type5-data’

[Fiona.Chen]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

[wei.zhang12]

Jetson
Deepstream 6.3
JetPack 5.1.2
TensorRT 8.5.2

[Fiona.Chen]

You can use normal rtsp decoding pipeline (rtspsrc->rtpdepay->h264parse->nvv4l2decoder->…) instead of using “nvurisrcbin”

[wei.zhang12]

Firstly, I need process multiple rtsp streams
Secondly, on Jetson device, nvv4l2decoder can’t support “extract-sei-type5-data”

[Fiona.Chen]

1. Even with nvurisrcbin, you need to get SEI info stream by stream.
2. On Jetson, please use GStreamer APIs. How to get the SEI payload in Deepstream pipeline - #3 by Fiona.Chen

[wei.zhang12]

@Fiona.Chen
That’s still no clear for me.
I am using the deepstream_test_3.py.
I don’t know how to use GStream APIs to extract SEI payload.
Do you have a piece of sample code for this?

Some pre-conditions for my system are

1. We are using deepstream-python
2. We are on Jetson device
3. We have multi rtsp streams to be processed

That’s why I am using deepstrea_test_3.py as a template.
If there is a way to modify some code inside deepstream_test_3.py to extract SEI payload, it will be great.

If we can’t archive this target by deepstream_test_3.py, please let me know how do I do this under deepstream framework.

[Fiona.Chen]

@wei.zhang12

I don’t know how to use GStream APIs to extract SEI payload.

The SEI header parsing does not need hardware resources. It has nothing to do with GPU, you can get the header data from the stream even in other platforms with the open source GStreamer APIs. The H264 spec is open standard( H.264 : Advanced video coding for generic audiovisual services (itu.int)). You may raise request to the GStreamer community for the API usages.

[Fiona.Chen]

There are GStreamer community: GStreamer: open source multimedia framework

Seems you are using python, the Python GStreamer Tutorial (brettviren.github.io) may be helpful.

[wei.zhang12]

I add a prob for rtph264depay and in the rtph264_src_pad_buffer_probe, I can extract SEI data. And this SEI only existed in Dahua Camera.

def parse_ntp_timestamp_from_sei(data):
    start_code_1 = -1
    start_code_2 = -1
    start_code_3 = -1
    start_code_4 = -1
    index = 0
    while index < len(data):
        start_code_1 = data[index]
        if start_code_1!=0:
            index+=1
            continue
        start_code_2 = data[index+1]
        start_code_3 = data[index+2]
        start_code_4 = data[index+3]
        if start_code_1 == 0 and start_code_2 == 0 and start_code_3 == 0 and start_code_4 > 0:
            nal_type = data[index+4]
            sei_payload_type = data[index+5]
            if nal_type==6 and sei_payload_type ==5:
                uuid = data[index+7:index+7+16].decode("utf-8")
                timestamp = data[index+23:index+23+13].decode("utf-8")
                return timestamp
        index+=1
    return ""

def rtph264_src_pad_buffer_probe(pad, info, u_data):
    print("rtph264_src_pad_buffer_probe")
    buf = info.get_buffer()
    if not buf:
        print("Unable to get GstBuffer for rtph264_src_pad_buffer_probe")
        return
    (result, mapinfo) = buf.map(Gst.MapFlags.READ)
    content =mapinfo.data
    timestamp = parse_ntp_timestamp_from_sei(content)
    print("SEI Timestamp:"+timestamp)
    return Gst.PadProbeReturn.OK
def decodebin_child_added(child_proxy, Object, name, user_data):
    print("Decodebin child added:", name, "\n")
    if name.find("decodebin") != -1:
        Object.connect("child-added", decodebin_child_added, user_data)
    if name.find("rtph264depay") != -1:
        print("find child rtph264depay")
        rtph264pad = Object.get_static_pad("src")
        rtph264pad.add_probe(Gst.PadProbeType.BUFFER, rtph264_src_pad_buffer_probe, 0)
		
    if ts_from_rtsp:
        if name.find("source") != -1:
            pyds.configure_source_for_ntp_sync(hash(Object))

[Fiona.Chen]

Thank you for sharing the code, it may be helpful for other users!