How can I improve instability of FPS when I use RTSP stream

[masaki_yamagishi]

• Hardware Platform: Jetson
• DeepStream Version: 6.4
• JetPack Version: 6.0DP
• TensorRT Version: 8.6
• Issue Type: questions

Please give me some instructions to improve instability of FPS when I use RTSP stream on DeepStream Application.
Especially, please tell me how I can integrate NTP timestamp from RTSP stream on DeepStream.

When I use RTSP stream from a camera(Tapo C510W) as source of DeepStream, FPS was unstable, like stop, go, and stop…

I omitted nvinfer, or nv3dsink, but there were no improvement.
Also, I tried to such as setting “sync=0” in sink section, I couldn’t get better result either.

I suppose this will be solved to use NTP timestamp, but I don’t know how I can do it.
As below, I confirmed the camera send Sender Reports.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_NTP_Timestamp.html
Altough I set “attach-sys-ts-as-ntp=0” or “1” in streammux section, there were no improvement.

There was no problem connecting to the camera.
I confirmed it worked fine by ffplay command.

I attached config file I used.
for_submit.txt (3.7 KB)

Best regards,

[fanzh]

1. could you set a local video in for_submit.txt to check if it is related to source?
2. could you share the result of the following command-line? it will show the actual fps by “current:”.

gst-launch-1.0  rtspsrc location=rtsp://127.0.0.1:8554/test ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink=fakesink  text-overlay=false sync=false -v |grep current

[masaki_yamagishi]

Thank you for your reply.

I tried tasks of your suggestion.

1. I set URI a local file, and it worked fine (as the attached picture).
   

   

2. I ran your command, and the result is below.

ryoyo@ubuntu:~$ gst-launch-1.0 rtspsrc location=rtsp://[appropriate config] ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink=fakesink text-overlay=false sync=false -v | grep current
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 9, dropped: 0, current: 16.65, average: 16.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 29, dropped: 0, current: 13.87, average: 14.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 37, dropped: 0, current: 14.72, average: 14.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 57, dropped: 0, current: 14.55, average: 14.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 64, dropped: 0, current: 13.02, average: 14.42
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 85, dropped: 0, current: 14.57, average: 14.46
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 93, dropped: 0, current: 14.79, average: 14.48
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 113, dropped: 0, current: 13.90, average: 14.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 121, dropped: 0, current: 14.81, average: 14.40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 141, dropped: 0, current: 14.50, average: 14.42
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 148, dropped: 0, current: 13.01, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 169, dropped: 0, current: 14.56, average: 14.37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 177, dropped: 0, current: 14.82, average: 14.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 197, dropped: 0, current: 13.88, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 205, dropped: 0, current: 14.83, average: 14.36
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 225, dropped: 0, current: 14.49, average: 14.37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 232, dropped: 0, current: 13.01, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 253, dropped: 0, current: 14.55, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 261, dropped: 0, current: 14.78, average: 14.35
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 281, dropped: 0, current: 13.90, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 289, dropped: 0, current: 14.84, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 309, dropped: 0, current: 14.49, average: 14.35
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 316, dropped: 0, current: 13.02, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 337, dropped: 0, current: 14.56, average: 14.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 345, dropped: 0, current: 14.79, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 365, dropped: 0, current: 13.90, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 373, dropped: 0, current: 14.78, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 393, dropped: 0, current: 14.50, average: 14.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 400, dropped: 0, current: 13.01, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 421, dropped: 0, current: 14.56, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 429, dropped: 0, current: 14.83, average: 14.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 449, dropped: 0, current: 13.89, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 457, dropped: 0, current: 14.74, average: 14.32

As a whole, the performance was fine.
I think my difficulty is caused by using RTSP stream as a source.

Best regards,

[fanzh]

Thanks for the sharing!

@masaki_yamagishi

current: 14.74, average: 14.32

please make sure you are testing the same RTSP source. from the log above, the instant fps and average fps got from the gst-launch are all about 15. which is almost same with the fps tested from ffplay. from the first screenshot, the instant fps is unstable while the average fps is stable. seems something causes the abnormal instance fps calculation.
using the same configuration file, I can’t reproduce this issue on Orin. could you help to narrow this issue by the following steps.

1. disable tracker and osd to check if the instant fps is stable.
2. use the following pipeline with nvstreammux and nvdsosd to check if the instant fps is unstable.

 gst-launch-1.0  rtspsrc location=rtsp://10.19.225.238:8554/test ! rtph264depay ! h264parse ! decodebin ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 !  nvdsosd ! fpsdisplaysink video-sink=fakesink  text-overlay=false sync=false -v |grep current

3. fps calculation is opensource. can you add log to print buffer_cnt[i], time2, str1->total_buffer_cnt, time1. wondering if the date is correct. after modified, please rebuild the deepstream-app in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app, then use ./deepstream-app to start.

      perf_struct.fps[i] = buffer_cnt[i] / time2;
      perf_struct.fps_avg[i] = str1->total_buffer_cnt / time1;

the whole path is \opt\nvidia\deepstream\deepstream-6.4\sources\apps\apps-common\src\deepstream_perf.c.

[fanzh]

could you also share a longer deepstram-app log? wondering if the instant fps becomes stable over time.

[masaki_yamagishi]

Sorry, I’m too busy this week, please wait until next week.
BTW, what do you mean “deepstream-app log”?

could you also share a longer deepstram-app log? wondering if the instant fps becomes stable over time.

Do you need longer fps record?

Best regards,

[fanzh]

Sorry, noticing you are testing deepstream-app, could you share a whole log with a longer time? one minute is enough, wondering if there is any abnormal printing and the instant fps will become stable over time.

[masaki_yamagishi]

Sorry to keep you waiting.

The essential issue is stacking (unstable) output when using IP cam as source.
Ultimately, the problem I want to solve is not to make same number of fps, but be stable output.
The phenomenon is like below.

In same environment, when using ffplay, the output is fine.

So, I think this stacking is depend on DeepStream framework.
How can I solve this?

1. use the following pipeline with nvstreammux and nvdsosd to check if the instant fps is unstable.

The log of that command is below.
gst_rtsp_log.txt (16 KB)

could you also share a longer deepstram-app log? wondering if the instant fps becomes stable over time.

The log of that command is below.

• disabled osd and with fakesink
  ds_rtsp_log.txt (3.5 KB)
• enabled osd and with nv3dsink
  ds_rtsp_log_eglsink.txt (3.6 KB)

3. fps calculation is opensource. can you add log to print buffer_cnt[i], time2, str1->total_buffer_cnt, time1. wondering if the date is correct. after modified, please rebuild the deepstream-app in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app, then use ./deepstream-app to start.

Sorry, I’ve not tried it yet.

As additional information, DeepStream integrated Metropolis Microservices for Jetson shows same issue.

Best regards,

[fanzh]

@masaki_yamagishi

disabled osd and with fakesink

Thanks for the sharing! seems using “disable osd and with fakesink” the video still stuttered. we still need to narrow down this issue. Thanks!

1. please test this simplest configuration file with deepstream-app? it only includes streammux and fakesink. source1_rtsp_dec_infer_resnet_int8_fan.txt (3.0 KB)
2. if still stutter, please use this faq
   to check which element will consume too much time.

[masaki_yamagishi]

Hi, @fanzh

Thank you for your kindly suggestion.

#1

please test this simplest configuration file with deepstream-app? it only includes streammux and fakesink.

I tried it and attached output of this.
ds_stuttered.txt (696 Bytes)
It seems stable.

#2

1. if still stutter, please use this faq
   to check which element will consume too much time.

I set env as below, and measured latencies of frames and each elements.

export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1
export NVDS_ENABLE_LATENCY_MEASUREMENT=1

I used below config file, which was customized from “source1_usb_dec_infer_resnet_int8.txt”, and was changed only source0.
rtsp_test.txt (3.9 KB)
The output is below.
latency_log.txt (303.2 KB)
Then, I organized that output, and made statistics.

This shows growing of latencies were occurred periodically.

And statistic values of each latencies are below.

• Frame Latency (ms)
  Max: 230.290039
  Min: 7.488037
  Avg: 69.22920974
  Std Dev: 65.64684938
• nvv4l2decoder0
  Max: 0.562012
  Min: 0.177979
  Avg: 0.357970157
  Std Dev: 0.07368124
• nvstreammux-src_bin_muxer
  Max: 113.719971
  Min: 0.655029
  Avg: 7.973373157
  Std Dev: 10.99303953
• primary_gie
  Max: 18.61792
  Min: 2.196045
  Avg: 6.51779423
  Std Dev: 3.571083002
• tiled_display_tiler
  Max: 14.291992
  Min: 1.331055
  Avg: 6.101954581
  Std Dev: 2.907566536
• osd_conv
  Max: 11.875977
  Min: 0.543945
  Avg: 2.78440003
  Std Dev: 1.991351875
• nvosd0
  Max: 18.539062
  Min: 0.00293
  Avg: 0.529278233
  Std Dev: 1.21280866

I wonder if nvstreammux was bottleneck relatively from Std Dev value, but it cannot be sure.
Do you have any opinion about that?

Best regards,

[fanzh]

PERF: FPS 0 (Avg)
PERF: 0.00 (0.00)
PERF: 19.33 (14.65)
PERF: 13.55 (15.42)
PERF: 15.09 (14.18)

1. from the log latency_log.txt, the fps is stable at 15. you can let it run for about 2 minutes to check if the fps is still stable.
2. noticing the camera 's fps is 14.25 in the issue description, please set batched-push-timeout=71000, 40000 is for fps 25. please refer to “solution 3” in this link.

[masaki_yamagishi]

Hi, @fanzh

Thank you for your reply.

I tried below and attached results.

1. from the log latency_log.txt, the fps is stable at 15. you can let it run for about 2 minutes to check if the fps is still stable.

I executed your sample config file around 20 minutes.
ds_stuttered_20240512.txt (5.9 KB)
It seems there was cycle of value of FPS, 17 fps and 11 fps.
But I don’t know whether that cycle implied the symptom that I’ve inquired.

2. noticing the camera 's fps is 14.25 in the issue description, please set batched-push-timeout=71000, 40000 is for fps 25. please refer to “solution 3” in this link.

I adjusted the config as “batched-push-timeout=71000” and ran with it.

But there were no improvement.

Could you tell more instruction to improve this stutter?

Best regards,

[fanzh]

from the ds_stuttered_20240512.txt, the fps still fluctuate from 11 to 17.

1. wondering if fps 11-17 will cause this severe stuttering, please test this commnad-line. it will show the video and calculate fps. please share the fps log and check if the video stutter.

  gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=false -v | grep current

2. In the point 1, if the video does not stutter, please use the following the command-line to check streammux cause the severe stuttering. it adds streammux based on the pipeline in point 1. please share the fps log and check if the video stutter.

gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 ! rtph264depay ! h264parse ! decodebin  ! mux.sink_0  nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=false -v | grep current

3. from your latest video, it still stutter. could you share the testing configuration file and log? Thanks!

[masaki_yamagishi]

Hi, @fanzh

Thank you for your dedicated reply.
I tried #1, and sink still was stuttered as below.

1. wondering if fps 11-17 will cause this severe stuttering, please test this commnad-line. it will show the video and calculate fps. please share the fps log and check if the video stutter.
   gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=false -v | grep current

The output of that command is attached.
ds_stuttered_20240515.txt (4.5 KB)

The value of FPS was stabled in this case, but sink wasn’t improved.
Thus, value of FPS, and even streammux are not relative to this issue.

I’m feeling we’re close to cause of this issue, but I don’t have concrete idea what the cause is and to improve it.
Could you give me instruction what should I do?

Thank you for your kindly supprt.

Best regards,

[fanzh]

To narrow down this issue, could you do the following three tests? Thanks!

1. could you share one minute timestamp log by this cmd?

ffprobe rtsp://address -show_frames    >timestamp.log

2. could you provide one minute stream data by this cmd?

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! mux. mpegtsmux name=mux ! filesink location=output.ts

3. to rule out the DeepStream issue, could you check if the out video stutter by this cmd which does not include any DeepStream plugin?

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! avdec_h264 ! fpsdisplaysink

if can’t play, please try ximagesink or xvimagesink.
if video does not stutter, please try this cmd, wondering if it is related to HW decoding.

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder  ! nv3dsink

[masaki_yamagishi]

Hi, @fanzh

Sorry for my late reply.

I attached results of your instructions.

1. could you share one minute timestamp log by this cmd?
   ffprobe rtsp://address -show_frames >timestamp.log

timestamp.log (2.1 MB)

2. could you provide one minute stream data by this cmd?
   gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! mux. mpegtsmux name=mux ! filesink location=output.ts

output.zip (1.1 MB)

3. to rule out the DeepStream issue, could you check if the out video stutter by this cmd which does not include any DeepStream plugin?
   gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! avdec_h264 ! fpsdisplaysink

if video does not stutter, please try this cmd, wondering if it is related to HW decoding.
gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink

There were no stuttered (and still remain issue of outset, with default settings of DeepStream).

Do you have any idea?

Best regards,

[fanzh]

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink

Thanks for the sharing! from your test result, the gst-launch cmd above run well. we can add other plugins step by step.

1. I only added the sterammux pluguin in to the cmd. please run it to check if the video stutters.

gst-launch-1.0 rtspsrc location=rtsp://xxx  ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout = 40000 live-source=1 !  nv3dsink

2. if the cmd in the step 1 run well. you can run deepstream-app with the same parameters. rtspsrc’ drop_on_latency is hardcoded to TRUE, please modify it to FALSE in create_rtsp_src_bin of opt\nvidia\deepstream\deepstream\sources\apps\apps-common\src\deepstream_source_bin.c, here is the code:

  g_object_set (G_OBJECT (bin->src_elem), "drop-on-latency", FALSE , NULL);
 printf("drop-on-latency = FALSE\n");

then back up the deepstream-app.

cp /opt/nvidia/deepstream/deepstream/bin/deepstream-app /opt/nvidia/deepstream/deepstream/bin/deepstream-app_ori

then make and make install in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app to make a new depstream-app. then run deepstream-app with this cfg to check if the video stutters.source1_rtsp_dec_infer_resnet_int8_streammux_fan.txt (3.0 KB). please make sure there will be “drop-on-latency = FALSE” log.

[masaki_yamagishi]

Hi, @fanzh

Sorry for my late reply because I was too busy in this May.

I tried below command, a stuttering was shown.

I only added the sterammux pluguin in to the cmd. please run it to check if the video stutters.
gst-launch-1.0 rtspsrc location=rtsp://xxx ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout = 40000 live-source=1 ! nv3dsink

Did it prove nvstreammux plugin is a cause of stuttering when one use a RTSP stream as source?
If it is true, how can I improve this stuttering?

Best regards,

[masaki_yamagishi]

Hi, @fanzh

Do you have any update?

Best regards,

[fanzh]

@masaki_yamagishi

Did it prove nvstreammux plugin is a cause of stuttering when one use a RTSP stream as source?

sorry for the late reply! from the latest test results, I suppose sometimes nvstreammux consumed too much time.

1. could you provide the nvstreammux 1.log by the following cmd. wondering if there is abnormal printings.

export GST_DEBUG=nvstreammux:6  &&  gst-launch-1.0 rtspsrc location=rtsp://address     ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 live-source=1 !  fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=true -v  >1.log 2>1.log

2. could you provide the nvstreammux 2.log by the following cmd. I increased the bufferpool size.

export GST_DEBUG=nvstreammux:6  &&  gst-launch-1.0 rtspsrc location=rtsp://address  ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000  live-source=1 buffer-pool-size=10  !  fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=true -v  >2.log 2>2.log