NTP timestamp (RTSP Sender Report) is not set to NvDsAudioFrameMeta::ntp_timestamp in my pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) dGPU
• DeepStream Version 6.1.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I am testing ntp timestamp (from RTSP sender report) in audio pieline.
I got NvDsAudioFrameMeta::npt_timestamp=0 in my custom pipeline.

1) I ran deepstream-audio with “attach-sys-ts-as-ntp=0” and could get correct ntp timestamp from RTSP sender report.

GST_DEBUG=nvstreammux_ntp:5
../deepstream-audio -c ds_audio_sonyc_rtsp_test_config.txt 2>&1  | grep -i ntp

0:00:26.782460284  1448 0x7f38d40265e0 DEBUG        nvstreammux_ntp gstnvstreammux_ntp.cpp:187:apply_correction_if_needed_rtcp:<src_bin_muxer> Frame rate based NTP timestamp correction applied.
0:00:26.782470088  1448 0x7f38d4003120 DEBUG        nvstreammux_ntp gstnvstreammux_ntp.cpp:187:apply_correction_if_needed_rtcp:<src_bin_muxer> Frame rate based NTP timestamp correction applied.
0:00:26.782507663  1448 0x7f38d4003120 INFO         nvstreammux_ntp gstnvstreammux_ntp.cpp:269:gst_nvds_ntp_calculator_get_buffer_ntp:<src_bin_muxer> Frame NTP calculated. Correction required.mode: RTCP SR. source 0: PTS:0:00:23.526679105(23526679105) NTP: 2023-03-24T11:16:21.644Z(1679656581644254392). NTP diff:-40264523975. Corrected ntp: 2023-03-24T11:17:01.977Z(1679656621977045033). ntp_time_epoch_ns = 2023-03-24T11:16:02.454Z(1679656562454999999) ntp_frame_timestamp = 0:00:04.337424712(4337424712) System Time: 2003-03-29T11:16:21.628Z Avg Frame Time: 0:00:00.068266666 Avg. Frame Rate: 14.65
0:00:26.782517693  1448 0x7f38d40265e0 INFO         nvstreammux_ntp gstnvstreammux_ntp.cpp:269:gst_nvds_ntp_calculator_get_buffer_ntp:<src_bin_muxer> Frame NTP calculated. Correction required.mode: RTCP SR. source 1: PTS:0:00:23.526687491(23526687491) NTP: 2023-03-24T11:16:21.644Z(1679656581644254392). NTP diff:-40264523975. Corrected ntp: 2023-03-24T11:17:01.977Z(1679656621977045033). ntp_time_epoch_ns = 2023-03-24T11:16:02.454Z(1679656562454999999) ntp_frame_timestamp = 0:00:04.337433098(4337433098) System Time: 2003-03-29T11:16:21.628Z Avg Frame Time: 0:00:00.068266666 Avg. Frame Rate: 14.65
### label:[ 2_machinery-impact ] source_id:[1] ntp 1679656621977045033 buf-pts 147579740436532640
### source_id:[1] ntp 1679656621977045033 buf-pts 147579740436532640
### label:[ 2_machinery-impact ] source_id:[0] ntp 1679656621977045033 buf-pts 147579740436532640
### source_id:[0] ntp 1679656621977045033 buf-pts 147579740436532640

“label: …” is a debug print which prints NvDsAudioFrameMeta::ntp_timestamp, which matches to nvstreammux_ntp debug print info.

2) In my pipeline, ntp_timestamp is always 0 and buf_pts seems to be “PTS”. Is there any hidden condition so that ntp_timestamp is properly written? I could not debug since streammux-new is closed source. I would appreciate any hints or suggestion. Thanks.

0:00:19.967665857   267 0x56013eff11e0 DEBUG        nvstreammux_ntp gstnvstreammux_ntp.cpp:187:apply_correction_if_needed_rtcp:<src_bin_muxer_audio> Frame rate based NTP timestamp correction applied.
0:00:19.967714101   267 0x56013eff11e0 INFO         nvstreammux_ntp gstnvstreammux_ntp.cpp:269:gst_nvds_ntp_calculator_get_buffer_ntp:<src_bin_muxer_audio> Frame NTP calculated. Correction required.mode: RTCP SR. source 0: PTS:0:00:09.216094839(9216094839) NTP: 2023-03-24T11:06:31.521Z(1679655991521093749). NTP diff:-11212810408. Corrected ntp: 2023-03-24T11:06:42.800Z(1679656002800037490). ntp_time_epoch_ns = 2023-03-24T11:06:25.958Z(1679655985958999999) ntp_frame_timestamp = 0:00:03.654001089(3654001089) System Time: 2023-03-24T11:06:31.505Z Avg Frame Time: 0:00:00.066133333 Avg. Frame Rate: 15.12
### source_id:[0] ntp 0 buf-pts 9216094839

“### source_id:[0] ntp 0 buf-pts 9216094839” is from printf() in the probe function attached to src of streammux(new) element.

My pipeline is like (rtspsrc → decodebin … ->streammux-new → nvinferserver → … → sink )
input was dynamically added after streammux-new is played.

Thanks.

3

Why do you testing ntp timestamp from RTSP SR? What is your pipeline’s sink?

4

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_FAQ.html#how-do-i-configure-the-pipeline-to-get-ntp-timestamps

5

Why do you testing ntp timestamp from RTSP SR?

I want to use ntp timestamp (from rtcp SR) in my product (audio pipeline).

What is your pipeline’s sink?

Customized msgbroker.
But tihs problem will reproduce with fakesink
because I debug printed at “src” of nvstreammux-new

Frequently Asked Questions — DeepStream 6.2 Release documentation

I already set it to false.

6

If you want to get the original ntp from rtcp SR, please access the rtcp information from rtcp. E.G. gst_rtcp_packet_sr_get_sender_info() interface GstRTCPBuffer. You may need to construct rtp session without rtspsrc. Please refer to GStreamer. rtpsession (gstreamer.freedesktop.org)

7

If you want to get the original ntp from rtcp SR,

I just want to use NTP timestamp feature of deepstream in my app.

My issue/question is

In my pipeline, ntp_timestamp is always 0 and buf_pts seems to be “PTS”. Is there any hidden condition so that ntp_timestamp is properly written? I could not debug since streammux-new is closed source. I would appreciate any hints or suggestion. Thanks.

8

Frequently Asked Questions — DeepStream 6.2 Release documentation (nvidia.com)

9

Even if I set System clock to 2003/3/, I got 2023year NPT log ( like this: NTP: 2023-03-24T11:06:31.521Z ).
So my app is indeed receiving SR

10

@Fiona.Chen

In my app, streammux(new) surely knows NTP time from SR:

0:00:25.982500533   474 0x55f4a69e6640 INFO         nvstreammux_ntp gstnvstreammux_ntp.cpp:269:gst_nvds_ntp_calculator_get_buffer_ntp:<src_bin_muxer_audio> Frame NTP calculated. Correction required.mode: RTCP SR. source 0: PTS:0:00:16.859157729(16859157729) NTP: 2023-04-03T07:15:25.011Z(1680506125011312499). NTP diff:-23948810238. Corrected ntp: 2023-04-03T07:15:49.026Z(1680506149026256070). ntp_time_epoch_ns = 2023-04-03T07:15:13.661Z(1680506113661999999) ntp_frame_timestamp = 0:00:05.509845229(5509845229) System Time: 2023-04-03T07:15:25.075Z Avg Frame Time: 0:00:00.066133333 Avg. Frame Rate: 15.12

But NvDsAudioFrameMeta::ntp_timestamp is 0 (probed at “src” of streammux).
I am 100% sure that attach-sys-ts-as-ntp=0 is set because the above ntp log does not appear when I set attach-sys-ts-as-ntp=1.

Could you please check the source code to see if there are conditions under which ntp_timestamp is not set?

11

@Fiona.Chen
When debuggin in DeepStream 6.2 instead of 6.1.1,
following message appears in my app.
Maybe related?

0:00:26.417152042  1451 0x55f967bb4300 DEBUG        nvstreammux_ntp gstnvstreammux_ntp.cpp:185:apply_correction_if_needed_rtcp:<src_bin_muxer_audio> Frame rate based NTP timestamp correction is disabled.
12

@Fiona.Chen
Ping.
When the “Frame rate based NTP timestamp correction is disabled.” message appear?

13

@Fiona.Chen ping

14

@Fiona.Chen May I have some support?

15

Please check your RTSP source. The ntp comes from the rtsp rtcp SR, it is not generated by DeepStream.

16

@Fiona.Chen

When the “Frame rate based NTP timestamp correction is disabled.” message appear?

To summarize what I know:

  • The gstreamer pipeline correctly recognizes the RTSP SR time (confirmed by logs)
  • I confirmed attach-sys-ts-as-ntp=0 is set in my app
  • But ntp_timestamp is 0 in my app
  • When ntp_timestamp is 0, I get the message “Frame rate based NTP timestamp correction is disabled.”
17

This means the received framerate is invalid.

18

@Fiona.Chen When the recieved framerate is invliad? What is the condition?

19

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

It is caused by the server side. You need to check the server.

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.