Can't push metadata in RTSP stream using deepstream-app

[theelitepro0224]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) - Jetson Orin AGX
• DeepStream Version - 7.0
• JetPack Version (valid for Jetson only) - 6.0
• TensorRT Version - 8.6.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
I am trying to add metadata along with the video to the RTSP output generated by deepstream-app.

1. I have modified create_udpsink_bin() and start_rtsp_streaming() functions in deepstream_sink_bin.c file.
2. I am using a custom plugin rtponvifmeta (please see capabilities below) and sending JSON sample metadata from create_udpsink_bin function.
3. Data flow path for metadata is metasrc → rtponvifmeta → udpsink at 5001 port under create_udpsink_bin.
4. In start_rtsp_streaming, udpsrc at port 5001 → rtsp server. To achieve this I have modified below

  sprintf (udpsrc_pipeline,
      "( udpsrc name=pay0 port=%d buffer-size=%lu caps=\"application/x-rtp, media=video, "
      "clock-rate=90000, encoding-name=%s, payload=96 \" )"
      "( udpsrc name=pay1 port=%d buffer-size=%lu caps=\"application/x-rtp, media=application, "
        "clock-rate=1000, encoding-name=VND.ONVIF.METADATA \" )",
      updsink_port_num, udp_buffer_size, encoder_name, updsink_port_num+1, udp_buffer_size);

I confirmed that data is received on pay1 using a pad probe callback of type GST_PAD_PROBE_TYPE_BUFFER. However metadata is not received on RTSP url. Using ffmpeg, I can see the two streams as below, and I can play video in VLC on a remote PC using rtsp link generated by deepstream-app.

Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080, 30 fps, 30 tbr, 90k tbn, 180k tbc
  Stream #0:1: Data: none

rtponvifmeta caps

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      application/onvif-metadata
                 parsed: true
  
  SRC template: 'src'
    Availability: Always
    Capabilities:
      application/x-rtp
                  media: { (string)application }
             clock-rate: 1000
          encoding-name: VND.ONVIF.METADATA

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'
  SINK: 'sink'
    Pad Template: 'sink'

Can you please help me figure out why rtsp metadata is not received? How I can resolve this issue of metadata not received.

Thanks.

[fanzh]

DeepStream sends the encoded stream by udpsink. opensource gst-rtsp-server receives stream by udpsrc, then transmits to clients.
seems you want to add rtsp meta in rtspserver. This issue would be outside of DeepStream. please refer to rtspserver code.

[theelitepro0224]

Thanks for your response and sharing the link. I would like to know if the video and metadata has to be in sync to get the metadata in output rtsp stream.

Also from the link you have shared I found below statement at gst-rtsp-server/docs at master · GStreamer/gst-rtsp-server · GitHub.

The pipeline description should contain elements named payN, one for each
 stream (ex. pay0, pay1, ...). Also, for increased compatibility each stream
 should have a different payload type which can be configured on the payloader.

I added payload=98 for metadata as well as pt=98 and still I am not getting metadata in rtsp stream. However, when I try to fetch rtsp stream I get below warning in deepstream-app. Does this mean udpsrc is ignoring payload or pt and therefore I am not see metadata in rtsp output. How can I get rid of this warning.

(deepstream-app:770119): GLib-GObject-WARNING **: 01:15:24.486: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'

[fanzh]

@theelitepro0224

I added payload=98 for metadata as well as pt=98

gstudpsrc is Gstreamer opensource plugn, which has pt property. how did you set pt=98? if using h264 encoding, the default value should be 96. please refer to Gstreamer opensource rtph264pay.