RTSP latency does not work with NVSTREAMMUX

Hi,
There can be a quick solution for smooth playback. Could you check if the fps of all sources is same? If so, we can try by setting live-source=false. With this, timestamps of outgoing nvstreammux gst buffers will increase based on framerate of the source. If fps info is not available from caps, timestamps will increase as per 30 fps framerate.

@DaneLLL,

All my sources are IP cameras configured to stream 1920x1080 @ 25fps.

Running the above command with live-source=false,

gst-launch-1.0 \
  rtspsrc location=$RTSP_PATH1 latency=5000 ! decodebin ! mux.sink_0 \
  nvstreammux name=mux batch-size=1 batched-push-timeout=40000 width=1920 height=1080 live-source=false \
  ! nvoverlaysink

it will show the initial frame, then wait 5000ms (5sec) and start playing. Looking at the embedded timestamp, one can still see the fast-forward but at a slower rate. Once it catches up, after ~25 seconds, I start seeing the following console warnings and the display update is very slow:

WARNING: from element /GstPipeline:pipeline0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0:
There may be a timestamping problem, or this computer is too slow.

The led me to running the following command with sync=false on the sync,

gst-launch-1.0 \
  rtspsrc location=$RTSP_PATH1 latency=5000 ! decodebin ! mux.sink_0 \
  nvstreammux name=mux batch-size=1 batched-push-timeout=40000 width=1920 height=1080 live-source=false \
  ! nvoverlaysink sync=false

but this causes the stream to immediately start playing with the laggy/stuttering display, most notable on moving objects.

1 Like

Your statement made me investigate what the caps are. I have run the above gst-launch-1.0 commands using the -v flag, maybe there is another way to check the caps.

I have 8 older IP cameras from Provision-ISR, and one newer model from Hikvision. All are configured to stream 1920x1080 @ 25fps as mentioned previously. From what I can tell the framerate in the caps for the Provision-ISR cameras is not correct.

Herewith the verbose logs:

Please also take note that the timestamp warnings when running with live-source=false mentioned in my previous post, does not occur with the Hikvision camera.

A few questions on running with live-source=false:

  • For a single stream, if no framerate info is available from the caps, will the timestamps increase as per the mentioned 30fps, or per the equivalent fps from the batched-push-timeoutvalue
  • For multiple streams, where at least one has framerate info available in the caps, will the streammuxer then increase timestamps per that stream’s framrate, or the batched-push-timeout value?

Thanks for the assistance.

Hi,
Are you able to run sources in 1920x1080 @ 30fps? The quick solution may work better in 30fps.

Unfotuanately not, as my cameras are PAL 50Hz and provide max 25FPS streams. See: DS-2CD2025FWD-I.

Could you provide any timeline on when proper latency support will be added?

Any update?

Hi,
For now we only have a quick solution for 30fps case. Sources@25fps may not work properly in current release(s). We are checking this usecase and timeline is uncertain.

Hi @DaneLLL,

Having Deepstream, and specifcally nvstreammux not correctly supporting latencies on RTSP streams is a serious issue/shortcoming for any production system that is based and Deepstream for monitoring RTSP streams.

As confirmed by @Fiona.Chen on another serious bug I logged regarding SmartRecord, there is an upcoming release. Can I expect a support for this to be included in that release?

Hi @DaneLLL,

Any update/resolution to the confirmed issue?

Hi,
We are still evaluating this request. Currently it is pending and not included in next release.

@DaneLLL, thank you for the confirmation - at least we can manage our expectation. I would still ask that you prioritize the latency handling as this is crucial for proper RTSP support in Deepstream.

@jasonpgf2a, have you experienced the improper latency handling and “laggy” display issue? Any pointers on handling RTSP streams?

@jlerasmus @DaneLLL
I have experienced issues with choppiness in playback of recorded files (using smart record) before and also slight pauses in playback but all issues are now resolved.

The issues with choppiness were fixed by a combination of things such as updating to deepstream 5.0.1 and ensuring that sync is set to false on all sync elements and live-stream is true on streammux.

The slight pause in recorded files took me 6 months to resolve. I’d got to the point where I was just going to live with it and had exhausted all avenues.
But then. I was just fiddling around with camera settings.
I use hikvision cameras and they recommend to set the I-frame interval to be the same as the framerate.
So I run my cameras at 25fps which means the I-frame rate was always 25 as well.

I changed the I-frame rate to 50 ! Double the fps value and magically the pauses in my recorded files went away and I’ve never seen them again for about 2 months now.

No one can explain why this fixes it.

Basically video processing is a dark art. More art than science that is for sure - especially with Gstreamer.
So if you have issues you want to apply a machine learning style approach to fiddle with every single parameter in random fashions and check the results. ;-)

I’d love there to be a machine way to tell if a video file plays nice and smooth at correct fps because then I could automate the process…

With regards to the rtsp latency. I tried many settings from 0 to 5s and to be honest I never ever saw the slightest difference in my recorded files.

3 Likes

Thanks @jasonpgf2a for the feedback and advise. I will definitely double check my pipeline configuration.

Please note that I only see this on the live streams via HDMI output (EGL or Overlay sinks), this is not visible in the smart-record recordings. This makes sense as smart-record inherently does buffering (queuing).

Have you seen such issues on the live-display?

Regarding the latency, it was confirmed by by @DaneLLL that this is an issue with nvstreammux not supporting RTSP latencies correctly.

hello @jasonpgf2a , how do you increase FPS setting of camera to 50? Does it base on kind of type of camera?

@trild-vietnam

I set the I-frame interval to 50, no the fps .

But minimum of I-frame of my camera need equal to FPS. Is that so? How much your camera FPS?

Jason,Thanks for your message. I will certainly tune I-frame parameter for my post:

Hi,
Please upgrade to DeepStream 6.0 and set sync-inputs=true to nvstreammux. The laggy video playback should not be seen. Please give it a try.

1 Like

Thanks for letting me know. I will setup and test using Deepstream 6.0 and provide feedback.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.