RTCP timestamp delay

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

**• Hardware Platform (GPU) RTX 3060
**• DeepStream Version 6.3
• TensorRT Version
**• NVIDIA GPU Driver Version (valid for GPU only) CUDA12.0
• Issue Type (questions)

I am using the RTCP protocol to obtain the frame timestamp from a surveillance camera and then comparing this timestamp with the local timestamp on my machine. I have noticed that the difference between these timestamps keeps increasing over time. Could this be due to the recognition processing time being too long? Are there any configuration settings that can improve this situation?

However, I have noticed that even without performing object detection, the delay continues to increase over time.

3

If you don’t use DeepStream, will the following command line also increase the delay?

gst-launch-1.0 rtspsrc location="" ! fakesink

If it is about Deepstream delay, you can try the following document first

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_NTP_Timestamp.html

4

The delay still occurs.
I conducted a small test where I set Attach-sys-ts-as-ntp=0 to obtain the RTSP timestamp. However, when I changed the camera time, the obtained timestamp did not significantly change but continued counting as before. I had to restart the program for the timestamp to reflect the new camera time. Why is this happening? Isn’t the RTCP protocol supposed to obtain the current time from the camera?

5

1.How did you test it? Are you using deepstream-app ? attach-sys-ts-as-ntp only works with deepstream-app

  1. If you just play it like below, will the delay increase?
    gst-launch-1.0 uridecodebin uri="your rtsp" ！nveglglessink

3.try to set live-source property of nvstreammux to 1.

If you use deepstream-app, you will find it in the configuration file.

6
  1. Yes, I am using deepstream-app
  2. It seems like there is some delay.
  3. The settings are already configured.
  4. I am comparing the frame->timestamp with my current time and noticing that the difference is increasing. Does this mean the delay is getting worse?
7

Is there a big delay? Will it accumulate? The delay depends on your rtsp camera model, normally the playback delay is not accumulative.

You can point the rtsp camera at a real clock to determine the delay

If playback causes delay accumulation, this issue maybe not related with DeepStream.

8
  1. There is screen delay, but it doesn’t seem to be accumulating continuously. I discovered the delay by setting Attach-sys-ts-as-ntp=0 to obtain the RTSP stream’s timestamp. Comparing this with my computer’s time, I noticed that the delay was increasing. Both the monitor and the computer are synchronized with the same NTP server and are on the same network.
  2. I set Attach-sys-ts-as-ntp=0 to obtain the RTSP timestamp. However, when I changed the camera time, the obtained timestamp did not significantly change but continued counting as before. I had to restart the program for the timestamp to reflect the new camera time. Why is this happening? Isn’t the RTCP protocol supposed to obtain the current time from the camera?
9

1.Which timestamp are you comparing with? There are frame_meta->buf_pts and frame_meta->ntp_timestamp and GST_BUFFER_PTS

2.If you want to get the pipeline latency, please refer to this FAQ

I’m not sure how you measure the latency of deepstream-app, can you share the code?

10 
for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
	NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
	int64 current_timestamp = TimeUtils::currentTimeInMillis();
	int64 def_timestamp = current_timestamp- (frame_meta->ntp_timestamp/1000000)
	std::cout<< " def_timestamp: " << def_timestamp <<std::endl
  }

I compared the NTP timestamp of each frame with the current time and found that the time difference is getting larger and larger.

11

It’s an interesting question, I checked the code of nvstreammux.

I used deepstream-test3 as an example and got the results I expected

This is the patch

diff --git a/sources/apps/sample_apps/deepstream-test3/deepstream_test3_app.c b/sources/apps/sample_apps/deepstream-test3/deepstream_test3_app.c
index 4c2cf00..e9d46e4 100644
--- a/sources/apps/sample_apps/deepstream-test3/deepstream_test3_app.c
+++ b/sources/apps/sample_apps/deepstream-test3/deepstream_test3_app.c
@@ -17,6 +17,7 @@
 #include <string.h>
 #include <sys/time.h>
 #include <cuda_runtime_api.h>
+#include <inttypes.h>
 
 #include "gstnvdsmeta.h"
 #include "nvds_yml_parser.h"
@@ -101,6 +102,13 @@ tiler_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
                 num_rects++;
             }
         }
+
+        // GstClockTime
+        struct timeval tv;
+        gettimeofday(&tv, NULL);
+        uint64_t clock = tv.tv_sec * 1000 + tv.tv_usec / 1000;
+        uint64_t delta = clock - frame_meta->ntp_timestamp / 1000000;
+        printf("clock = %" PRIu64 " ntp = %" PRIu64 " delta: %" PRIu64 "ms\n", clock, frame_meta->ntp_timestamp, delta);
           g_print ("Frame Number = %d Number of objects = %d "
             "Vehicle Count = %d Person Count = %d\n",
             frame_meta->frame_num, num_rects, vehicle_count, person_count);
@@ -255,6 +263,7 @@ create_source_bin (guint index, gchar * uri)
     g_object_set (G_OBJECT (uri_decode_bin), "cudadec-memtype", 0, NULL);
   } else {
     uri_decode_bin = gst_element_factory_make ("uridecodebin", "uri-decode-bin");
+    configure_source_for_ntp_sync(uri_decode_bin);
   }
 
   if (!bin || !uri_decode_bin) {
@@ -445,7 +454,8 @@ main (int argc, char *argv[])
 #ifdef __aarch64__
       sink = gst_element_factory_make ("nv3dsink", "nvvideo-renderer");
 #else
-      sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
+      //sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
+      sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");
 #endif
     }
   }
@@ -532,6 +542,10 @@ main (int argc, char *argv[])
       }
   }
 
+  g_object_set (G_OBJECT (streammux), "live-source", 1, NULL);
+  g_object_set (G_OBJECT (streammux), "frame-duration", 0, NULL);
+  g_object_set (G_OBJECT (streammux), "attach-sys-ts", 0, NULL);
+
   /* we add a message handler */
   bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
   bus_watch_id = gst_bus_add_watch (bus, bus_call, loop);

If you want to use the ntp time in rtcp, you need to set both attach-sys-ts and frame-duration to 0
deepstream-app is similar.

This is my result. You can see that the delay has not increased.

TP: 2024-06-11T12:16:34.079Z(1718108194079700138). NTP diff:33364781. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.192Z
clock = 1718108194193 ntp = 1718108194079700138 delta: 114ms
0:00:15.579209610 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.607745284(6607745284) NTP: 2024-06-11T12:16:34.113Z(1718108194113064935). NTP diff:33364797. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.223Z
clock = 1718108194224 ntp = 1718108194113064935 delta: 111ms
0:00:15.613469316 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.641107884(6641107884) NTP: 2024-06-11T12:16:34.146Z(1718108194146427535). NTP diff:33362600. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.257Z
clock = 1718108194258 ntp = 1718108194146427535 delta: 112ms
0:00:15.651864745 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.674470515(6674470515) NTP: 2024-06-11T12:16:34.179Z(1718108194179790166). NTP diff:33362631. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.296Z
clock = 1718108194297 ntp = 1718108194179790166 delta: 118ms
0:00:15.680095551 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.707833179(6707833179) NTP: 2024-06-11T12:16:34.213Z(1718108194213152830). NTP diff:33362664. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.324Z
clock = 1718108194325 ntp = 1718108194213152830 delta: 112ms
0:00:15.713640264 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.741195875(6741195875) NTP: 2024-06-11T12:16:34.246Z(1718108194246515526). NTP diff:33362696. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.357Z
clock = 1718108194358 ntp = 1718108194246515526 delta: 112ms
0:00:15.755680945 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.774558602(6774558602) NTP: 2024-06-11T12:16:34.279Z(1718108194279878253). NTP diff:33362727. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.399Z
clock = 1718108194400 ntp = 1718108194279878253 delta: 121ms
0:00:15.799888705 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.807921362(6807921362) NTP: 2024-06-11T12:16:34.313Z(1718108194313241013). NTP diff:33362760. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.444Z
clock = 1718108194445 ntp = 1718108194313241013 delta: 132ms
0:00:15.819295893 32314 0x7f39f4001f40 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:259:gst_nvds_ntp_calculator_get_buffer_ntp:<stream-muxer> Frame NTP calculated. mode: RTCP SR. source 0: PTS:0:00:06.841284153(6841284153) NTP: 2024-06-11T12:16:34.346Z(1718108194346603804). NTP diff:33362791. ntp_time_epoch_ns = 2024-06-11T12:16:32.977Z(1718108192977646331) ntp_frame_timestamp = 0:00:05.472326680(5472326680) System Time: 2024-06-11T12:16:34.463Z
clock = 1718108194464 ntp = 1718108194346603804 delta: 118ms
12

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.