What's the detail implementation of function "configure_source_for_ntp_sync"

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Tesla T4
• DeepStream Version 5.0.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 10.2
• Issue Type( questions, new requirements, bugs) Questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I want to know what did deepstream do to the “gstrtspsrc” element in function “configure_source_for_ntp_sync”.

I have another gstreamer pipeline reading the same RTSP stream with DeepStream (RTCP SR already enabled). I want this two pipeline to have same ntp-timestamp so that I can syncronize the meta information like bounding boxes. FYI, my own pipeline looks like this:

gst-launch-1.0 rtspsrc location="rtsp://rtsp_stream:554/lab1" protocols=GST_RTSP_LOWER_TRANS_TCP ! rtph264depay ! h264parse ! queue ! splitmuxsink location="lab1-splitmuxsink_%05d.ts" muxer=mpegtsmux max-size-time=10000000000

However, I found that my pipeline’s gstreamer timestamp is drifting, i.e., the difference between multifilesink’s message’s timestamp value & gstjitterbuffer’s gstreamer time (gstreamer_time += ((sr_ext_rtptime - base_rtptime) * GST_SECOND / clock_rate)) upon “handle_sync” signal is steadily growing (from seconds to tens of seconeds).

My implementation of time syncronization between gsteamer & RTCP SR is obtained from DeepStream 4.0.1 SDK since this is the only open source version.

So I want to know how DeepSteram application deal with this time drifting internally between gstreamer buffer timestamp & RTCP SR’s gstreamer timestamp.

Please give me some instruction.

3

As I dig into the detail implementation with help of Gstreamer Dot graph debug option, the source pipeline of DeepStream 5.0.0 test5 app’s rtsp source is like this.

image
image3117×429 148 KB

The most important part is “GstRTPSrc” and “GstRtpJitterBuffer” element.

For GstRtpSrc, I set the following properties:

  /* we set the input rtsp to the source element */
  g_object_set (G_OBJECT (appCtx->source), "location", appCtx->uri, NULL);
  g_object_set (G_OBJECT (appCtx->source), "latency", 2000, NULL);
  g_object_set (G_OBJECT (appCtx->source), "drop-on-latency", TRUE, NULL);
  /** 
   * none (0) – Only use RTP timestamps
   * slave (1) – Slave receiver to sender clock
   * buffer (2) – Do low/high watermark buffering
   * auto (3) – Choose mode depending on stream live
   * synced (4) – Synchronized sender and receiver clocks
   * Follow GstRtpJitterBuffer property from DeepStream Pipeline.
  */
  g_object_set (G_OBJECT (appCtx->source), "buffer-mode", 4, NULL);

For GstRtpJitterBuffer, I set following properties:

static void
rtp_bin_new_jitter_buffer (GstBin  *rtpbin,
               GstElement *jitterbuffer,
               guint       session,
               guint       ssrc,
               gpointer    user_data)
{
  // Follow GstRtpJitterBuffer property from DeepStream Pipeline.
  g_object_set (G_OBJECT (jitterbuffer), "max-rtcp-rtp-time-diff", -1, NULL);
  g_object_set (G_OBJECT (jitterbuffer), "faststart-min-packets", 1, NULL);
  /** Request for the `handle-sync` signal
   * in jitterbuffer to lawfully tap
   * RTCP Sender Report
   */
  g_signal_connect(G_OBJECT(jitterbuffer), "handle-sync",
                   G_CALLBACK(rtp_bin_handle_sync),
                   user_data);
}

static void 
cb_rtsp_src_elem_added (GstBin *bin, GstElement *element, gpointer u_data) {
  if(strstr(GST_ELEMENT_NAME(element), "manager"))
  {
      /** RtpBin: Request new-jitterbuffer signal */
      g_signal_connect(G_OBJECT(element), "new-jitterbuffer",
                       G_CALLBACK(rtp_bin_new_jitter_buffer),
                       u_data);
  }
}

int
main (int   argc,
      char *argv[])
{
  ...
  /* rtcp sender report handler */
  g_signal_connect (G_OBJECT (appCtx->source), "element-added", G_CALLBACK(cb_rtsp_src_elem_added), appCtx);
  ...
}

Then the time drifting behavior seems disappear. May be this is related to GstRtpJitterBuffer’s “max-rtcp-rtp-time-diff” property. In order to ensure synchronization between sender & receiver, we have to rely solely on RTCP sender report, no matter how much the diff is.

1 Like