How can I improve instability of FPS when I use RTSP stream

• Hardware Platform: Jetson
• DeepStream Version: 6.4
• JetPack Version: 6.0DP
• TensorRT Version: 8.6
• Issue Type: questions

Please give me some instructions to improve instability of FPS when I use RTSP stream on DeepStream Application.
Especially, please tell me how I can integrate NTP timestamp from RTSP stream on DeepStream.

When I use RTSP stream from a camera(Tapo C510W) as source of DeepStream, FPS was unstable, like stop, go, and stop…

I omitted nvinfer, or nv3dsink, but there were no improvement.
Also, I tried to such as setting “sync=0” in sink section, I couldn’t get better result either.


I suppose this will be solved to use NTP timestamp, but I don’t know how I can do it.
As below, I confirmed the camera send Sender Reports.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_NTP_Timestamp.html
Altough I set “attach-sys-ts-as-ntp=0” or “1” in streammux section, there were no improvement.

There was no problem connecting to the camera.
I confirmed it worked fine by ffplay command.

I attached config file I used.
for_submit.txt (3.7 KB)

Best regards,

  1. could you set a local video in for_submit.txt to check if it is related to source?
  2. could you share the result of the following command-line? it will show the actual fps by “current:”.
gst-launch-1.0  rtspsrc location=rtsp://127.0.0.1:8554/test ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink=fakesink  text-overlay=false sync=false -v |grep current

Thank you for your reply.

I tried tasks of your suggestion.

  1. I set URI a local file, and it worked fine (as the attached picture).

  2. I ran your command, and the result is below.

ryoyo@ubuntu:~$ gst-launch-1.0 rtspsrc location=rtsp://[appropriate config] ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink=fakesink text-overlay=false sync=false -v | grep current
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 9, dropped: 0, current: 16.65, average: 16.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 29, dropped: 0, current: 13.87, average: 14.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 37, dropped: 0, current: 14.72, average: 14.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 57, dropped: 0, current: 14.55, average: 14.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 64, dropped: 0, current: 13.02, average: 14.42
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 85, dropped: 0, current: 14.57, average: 14.46
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 93, dropped: 0, current: 14.79, average: 14.48
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 113, dropped: 0, current: 13.90, average: 14.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 121, dropped: 0, current: 14.81, average: 14.40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 141, dropped: 0, current: 14.50, average: 14.42
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 148, dropped: 0, current: 13.01, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 169, dropped: 0, current: 14.56, average: 14.37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 177, dropped: 0, current: 14.82, average: 14.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 197, dropped: 0, current: 13.88, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 205, dropped: 0, current: 14.83, average: 14.36
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 225, dropped: 0, current: 14.49, average: 14.37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 232, dropped: 0, current: 13.01, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 253, dropped: 0, current: 14.55, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 261, dropped: 0, current: 14.78, average: 14.35
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 281, dropped: 0, current: 13.90, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 289, dropped: 0, current: 14.84, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 309, dropped: 0, current: 14.49, average: 14.35
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 316, dropped: 0, current: 13.02, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 337, dropped: 0, current: 14.56, average: 14.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 345, dropped: 0, current: 14.79, average: 14.34
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 365, dropped: 0, current: 13.90, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 373, dropped: 0, current: 14.78, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 393, dropped: 0, current: 14.50, average: 14.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 400, dropped: 0, current: 13.01, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 421, dropped: 0, current: 14.56, average: 14.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 429, dropped: 0, current: 14.83, average: 14.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 449, dropped: 0, current: 13.89, average: 14.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 457, dropped: 0, current: 14.74, average: 14.32

As a whole, the performance was fine.
I think my difficulty is caused by using RTSP stream as a source.

Best regards,

Thanks for the sharing!

please make sure you are testing the same RTSP source. from the log above, the instant fps and average fps got from the gst-launch are all about 15. which is almost same with the fps tested from ffplay. from the first screenshot, the instant fps is unstable while the average fps is stable. seems something causes the abnormal instance fps calculation.
using the same configuration file, I can’t reproduce this issue on Orin. could you help to narrow this issue by the following steps.

  1. disable tracker and osd to check if the instant fps is stable.
  2. use the following pipeline with nvstreammux and nvdsosd to check if the instant fps is unstable.
 gst-launch-1.0  rtspsrc location=rtsp://10.19.225.238:8554/test ! rtph264depay ! h264parse ! decodebin ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 !  nvdsosd ! fpsdisplaysink video-sink=fakesink  text-overlay=false sync=false -v |grep current
  1. fps calculation is opensource. can you add log to print buffer_cnt[i], time2, str1->total_buffer_cnt, time1. wondering if the date is correct. after modified, please rebuild the deepstream-app in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app, then use ./deepstream-app to start.
      perf_struct.fps[i] = buffer_cnt[i] / time2;
      perf_struct.fps_avg[i] = str1->total_buffer_cnt / time1;

the whole path is \opt\nvidia\deepstream\deepstream-6.4\sources\apps\apps-common\src\deepstream_perf.c.

could you also share a longer deepstram-app log? wondering if the instant fps becomes stable over time.

Sorry, I’m too busy this week, please wait until next week.
BTW, what do you mean “deepstream-app log”?

could you also share a longer deepstram-app log? wondering if the instant fps becomes stable over time.

Do you need longer fps record?

Best regards,

Sorry, noticing you are testing deepstream-app, could you share a whole log with a longer time? one minute is enough, wondering if there is any abnormal printing and the instant fps will become stable over time.

Sorry to keep you waiting.

The essential issue is stacking (unstable) output when using IP cam as source.
Ultimately, the problem I want to solve is not to make same number of fps, but be stable output.
The phenomenon is like below.

In same environment, when using ffplay, the output is fine.

So, I think this stacking is depend on DeepStream framework.
How can I solve this?

  1. use the following pipeline with nvstreammux and nvdsosd to check if the instant fps is unstable.

The log of that command is below.
gst_rtsp_log.txt (16 KB)

could you also share a longer deepstram-app log? wondering if the instant fps becomes stable over time.

The log of that command is below.

  1. fps calculation is opensource. can you add log to print buffer_cnt[i], time2, str1->total_buffer_cnt, time1. wondering if the date is correct. after modified, please rebuild the deepstream-app in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app, then use ./deepstream-app to start.

Sorry, I’ve not tried it yet.

As additional information, DeepStream integrated Metropolis Microservices for Jetson shows same issue.

Best regards,

Thanks for the sharing! seems using “disable osd and with fakesink” the video still stuttered. we still need to narrow down this issue. Thanks!

  1. please test this simplest configuration file with deepstream-app? it only includes streammux and fakesink. source1_rtsp_dec_infer_resnet_int8_fan.txt (3.0 KB)
  2. if still stutter, please use this faq
    to check which element will consume too much time.

Hi, @fanzh

Thank you for your kindly suggestion.

#1

please test this simplest configuration file with deepstream-app? it only includes streammux and fakesink.

I tried it and attached output of this.
ds_stuttered.txt (696 Bytes)
It seems stable.

#2

  1. if still stutter, please use this faq
    to check which element will consume too much time.

I set env as below, and measured latencies of frames and each elements.

export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1
export NVDS_ENABLE_LATENCY_MEASUREMENT=1

I used below config file, which was customized from “source1_usb_dec_infer_resnet_int8.txt”, and was changed only source0.
rtsp_test.txt (3.9 KB)
The output is below.
latency_log.txt (303.2 KB)
Then, I organized that output, and made statistics.


This shows growing of latencies were occurred periodically.

And statistic values of each latencies are below.

  • Frame Latency (ms)
    Max: 230.290039
    Min: 7.488037
    Avg: 69.22920974
    Std Dev: 65.64684938
  • nvv4l2decoder0
    Max: 0.562012
    Min: 0.177979
    Avg: 0.357970157
    Std Dev: 0.07368124
  • nvstreammux-src_bin_muxer
    Max: 113.719971
    Min: 0.655029
    Avg: 7.973373157
    Std Dev: 10.99303953
  • primary_gie
    Max: 18.61792
    Min: 2.196045
    Avg: 6.51779423
    Std Dev: 3.571083002
  • tiled_display_tiler
    Max: 14.291992
    Min: 1.331055
    Avg: 6.101954581
    Std Dev: 2.907566536
  • osd_conv
    Max: 11.875977
    Min: 0.543945
    Avg: 2.78440003
    Std Dev: 1.991351875
  • nvosd0
    Max: 18.539062
    Min: 0.00293
    Avg: 0.529278233
    Std Dev: 1.21280866

I wonder if nvstreammux was bottleneck relatively from Std Dev value, but it cannot be sure.
Do you have any opinion about that?

Best regards,

PERF: FPS 0 (Avg)
PERF: 0.00 (0.00)
PERF: 19.33 (14.65)
PERF: 13.55 (15.42)
PERF: 15.09 (14.18)

  1. from the log latency_log.txt, the fps is stable at 15. you can let it run for about 2 minutes to check if the fps is still stable.
  2. noticing the camera 's fps is 14.25 in the issue description, please set batched-push-timeout=71000, 40000 is for fps 25. please refer to “solution 3” in this link.

Hi, @fanzh

Thank you for your reply.

I tried below and attached results.

  1. from the log latency_log.txt, the fps is stable at 15. you can let it run for about 2 minutes to check if the fps is still stable.

I executed your sample config file around 20 minutes.
ds_stuttered_20240512.txt (5.9 KB)
It seems there was cycle of value of FPS, 17 fps and 11 fps.
But I don’t know whether that cycle implied the symptom that I’ve inquired.

  1. noticing the camera 's fps is 14.25 in the issue description, please set batched-push-timeout=71000, 40000 is for fps 25. please refer to “solution 3” in this link.

I adjusted the config as “batched-push-timeout=71000” and ran with it.


But there were no improvement.

Could you tell more instruction to improve this stutter?

Best regards,

from the ds_stuttered_20240512.txt, the fps still fluctuate from 11 to 17.

  1. wondering if fps 11-17 will cause this severe stuttering, please test this commnad-line. it will show the video and calculate fps. please share the fps log and check if the video stutter.
  gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=false -v | grep current
  1. In the point 1, if the video does not stutter, please use the following the command-line to check streammux cause the severe stuttering. it adds streammux based on the pipeline in point 1. please share the fps log and check if the video stutter.
gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 ! rtph264depay ! h264parse ! decodebin  ! mux.sink_0  nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=false -v | grep current
  1. from your latest video, it still stutter. could you share the testing configuration file and log? Thanks!

Hi, @fanzh

Thank you for your dedicated reply.
I tried #1, and sink still was stuttered as below.

  1. wondering if fps 11-17 will cause this severe stuttering, please test this commnad-line. it will show the video and calculate fps. please share the fps log and check if the video stutter.
    gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 ! rtph264depay ! h264parse ! decodebin ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=false -v | grep current

The output of that command is attached.
ds_stuttered_20240515.txt (4.5 KB)

The value of FPS was stabled in this case, but sink wasn’t improved.
Thus, value of FPS, and even streammux are not relative to this issue.

I’m feeling we’re close to cause of this issue, but I don’t have concrete idea what the cause is and to improve it.
Could you give me instruction what should I do?

Thank you for your kindly supprt.

Best regards,

To narrow down this issue, could you do the following three tests? Thanks!

  1. could you share one minute timestamp log by this cmd?
ffprobe rtsp://address -show_frames    >timestamp.log
  1. could you provide one minute stream data by this cmd?
gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! mux. mpegtsmux name=mux ! filesink location=output.ts
  1. to rule out the DeepStream issue, could you check if the out video stutter by this cmd which does not include any DeepStream plugin?
gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! avdec_h264 ! fpsdisplaysink

if can’t play, please try ximagesink or xvimagesink.
if video does not stutter, please try this cmd, wondering if it is related to HW decoding.

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder  ! nv3dsink

Hi, @fanzh

Sorry for my late reply.

I attached results of your instructions.

  1. could you share one minute timestamp log by this cmd?
    ffprobe rtsp://address -show_frames >timestamp.log

timestamp.log (2.1 MB)

  1. could you provide one minute stream data by this cmd?
    gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! mux. mpegtsmux name=mux ! filesink location=output.ts

output.zip (1.1 MB)

  1. to rule out the DeepStream issue, could you check if the out video stutter by this cmd which does not include any DeepStream plugin?
    gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! avdec_h264 ! fpsdisplaysink

if video does not stutter, please try this cmd, wondering if it is related to HW decoding.
gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink

There were no stuttered (and still remain issue of outset, with default settings of DeepStream).

Do you have any idea?

Best regards,

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink

Thanks for the sharing! from your test result, the gst-launch cmd above run well. we can add other plugins step by step.

  1. I only added the sterammux pluguin in to the cmd. please run it to check if the video stutters.
gst-launch-1.0 rtspsrc location=rtsp://xxx  ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout = 40000 live-source=1 !  nv3dsink
  1. if the cmd in the step 1 run well. you can run deepstream-app with the same parameters. rtspsrc’ drop_on_latency is hardcoded to TRUE, please modify it to FALSE in create_rtsp_src_bin of opt\nvidia\deepstream\deepstream\sources\apps\apps-common\src\deepstream_source_bin.c, here is the code:
  g_object_set (G_OBJECT (bin->src_elem), "drop-on-latency", FALSE , NULL);
 printf("drop-on-latency = FALSE\n");

then back up the deepstream-app.

cp /opt/nvidia/deepstream/deepstream/bin/deepstream-app /opt/nvidia/deepstream/deepstream/bin/deepstream-app_ori

then make and make install in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app to make a new depstream-app. then run deepstream-app with this cfg to check if the video stutters.source1_rtsp_dec_infer_resnet_int8_streammux_fan.txt (3.0 KB). please make sure there will be “drop-on-latency = FALSE” log.