Latency and video artifacts in multiple RTSP I/O

aieng · June 12, 2024, 10:08pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 5.1.1
• Issue Type( questions, new requirements, bugs) Question

Hello, I have an application which is capable of taking multiple RTSP sources as input and outputs multiple RTSP streams after performing inference / tracking etc. I have 2 HIKVISION AcuSense Network Camera and 2 DAHUA IR Dome Network Cameras. On utilizing only the 2 HIKVISION cameras RTSP streams as input for my application, the rtsp video output does not have any observable latency/buffering in the streams. However, when I use 1 HIKVISION/1 Dahua camera I am observing latency (frequent freezing/buffering) in the output RTSP stream. Same issue happens if I utilize 2 Dahua cameras. All configurations for the cameras sources are at 1280x720p at 25FPS. What could be possible reasons for this issue to happen?

On increasing the number of streams to 5-7 (With either cameras), I am also observing video artifacts being generated in the stream similar to the issue here however the rtspsrc properties suggested in the post do not help with resolving it. The only way I am able to resolve this issue is by increasing the drop-frame-interval of the nvv4l2decoder, however that leads to drop in my output video FPS which is not the preferred solution. Please advise.

fanzh · June 13, 2024, 2:52am

about the latency issue, which sample are you testing or referring to? could you share the whole media pipeline? please refer to this topic for nvstreammux setting.
about the artifacts issue, please rule out the RTSP receiving issue. You can enable rtspsrc debug message to check the RTSP problems.

aieng · June 13, 2024, 7:57pm

Hi @fanzh, this is a custom pipeline which builds on runtime addition of sources and multiple rtsp input / multiple rtsp output deepstream applications. I have tried utilizing the nvstreammux settings mentioned (for old streammux) in that topic and it does not help with the latency issue observed when using RTSP streams from different camera sources.

Regarding the artifacts issue, I have enabled rtspsrc debug logs by checking for GST_DEBUG=rtspsrc:7 also tried other levels of logs but they don’t indicate any errors in the logs. FYI - I have enabled “drop-on-latency” for the rtspsrc to be true. I have tested with it being false as well and it did help reduce the artifacts being generated however the output stream starts lagging behind the input stream significantly over a period of time. Can you please advise what to look for when identifying packet loss or if there is any specific element to check for? (I’m using uridecodebin to accept the rtsp input streams and then link it with the streammux). What could be other possible reasons for this issue?

fanzh · June 14, 2024, 9:11am

could you share the whole Gstreamer pipeline? what elements are used? You can use forum private email for the private information. please click forum avatar-> personal messages->new message.
using two Dahua camera has issue while using two HIKVISION camera has no issue. the only one difference is RTSP source. so please check the Dahua camera first. In the log with GST_DEBUG=rtspsrc:7, are there “missing packets” or “dropping old packet” printing? which means packets loss.
if only using one Dahua camera, will the app run well?

aieng · June 14, 2024, 4:53pm

could you share the whole Gstreamer pipeline? what elements are used? You can use forum private email for the private information. please click forum avatar-> personal messages->new message.

I will get back to you on this.

using two Dahua camera has issue while using two HIKVISION camera has no issue. the only one difference is RTSP source. so please check the Dahua camera first. In the log with GST_DEBUG=rtspsrc:7, are there “missing packets” or “dropping old packet” printing? which means packets loss.

Just to clarify - I see the video artifact issue happen irrespective of the camera utilized. This happens when I increase the number of streams being processed. The latency/buffering issue happens based on the camera type.

Regarding the video artifact issue, I was not able to see the packet loss logs with GST_DEBUG=rtspsrc:7. However on using GST_DEBUG=3,rtpjitterbuffer:6,v4l2videodec:6 as mentioned in this topic, I get the following logs when artifacts are visible in the output stream.

0:02:33.946446108   449 0xfffe2401c580 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2518:calculate_packet_spacing:<rtpjitterbuffer2> new packet spacing 0:00:00.033008575 old packet spacing 0:00:00.044952160 combined to 0:00:00.041966263
0:02:33.946457564   449 0xfffe2401c580 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3211:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer2> Queue full, dropping old packet 0xfffe640f22c0
0:02:33.946479901   449 0xfffe2401c580 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3268:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer2> Pushed packet #24722, now 221 packets, head: 0, percent -1
0:02:33.946531358   449 0xfffe74002aa0 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2983:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer3> Received packet #24722 at time 0:02:33.542100394, discont 0, rtx 0
0:02:33.946315417   449 0xfffe8c0041e0 LOG          rtpjitterbuffer gstrtpjitterbuffer.c:2687:calculate_jitter:<rtpjitterbuffer0> dtsdiff 0:00:00.020016972 rtptime 0:00:00.033000000, clock-rate 90000, diff 0:00:00.012983028, jitter: 0:00:00.008487163
0:02:33.946592351   449 0xfffe74002aa0 LOG          rtpjitterbuffer gstrtpjitterbuffer.c:2687:calculate_jitter:<rtpjitterbuffer3> dtsdiff 0:00:00.020229136 rtptime 0:00:00.033000000, clock-rate 90000, diff 0:00:00.012770864, jitter: 0:00:00.008343601
0:02:33.946600479   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3083:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer0> expected #24788, got #24788, gap of 0
0:02:33.946629728   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3113:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer0> Clearing gap packets
0:02:33.946649121   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:800:rtp_jitter_buffer_calculate_pts: extrtp 1633457616, gstrtp 5:02:29.529066666, base 4:59:56.056066666, send_diff 0:02:33.473000000
0:02:33.946660033   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:579:calculate_skew: time 0:02:33.541863333, base 0:00:00.072121920, recv_diff 0:02:33.469741413, slope 8
0:02:33.946665793   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:659:calculate_skew: delta -3258587, new min: -3316500
0:02:33.946671681   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:681:calculate_skew: skew -4008129, out 0:02:33.541113791
0:02:33.946681025   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2518:calculate_packet_spacing:<rtpjitterbuffer0> new packet spacing 0:00:00.033005578 old packet spacing 0:00:00.044953891 combined to 0:00:00.041966812
0:02:33.946689762   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3211:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer0> Queue full, dropping old packet 0xfffe64105980
0:02:33.946696130   449     0x12da0580 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2983:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer4> Received packet #24722 at time 0:02:33.542365169, discont 0, rtx 0
0:02:33.946703330   449 0xfffe8c0041e0 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3268:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer0> Pushed packet #24788, now 221 packets, head: 0, percent -1
0:02:33.946725123   449     0x12da0580 LOG          rtpjitterbuffer gstrtpjitterbuffer.c:2687:calculate_jitter:<rtpjitterbuffer4> dtsdiff 0:00:00.020272722 rtptime 0:00:00.033000000, clock-rate 90000, diff 0:00:00.012727278, jitter: 0:00:00.008375080
0:02:33.946736963   449     0x12da0580 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3083:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer4> expected #24722, got #24722, gap of 0
0:02:33.946741795   449     0x12da0580 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:3113:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer4> Clearing gap packets
0:02:33.946751587   449     0x12da0580 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:800:rtp_jitter_buffer_calculate_pts: extrtp 1633454646, gstrtp 5:02:29.496066666, base 5:00:41.648066666, send_diff 0:01:47.848000000
0:02:33.946761187   449     0x12da0580 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:579:calculate_skew: time 0:02:33.542365169, base 0:00:45.726063478, recv_diff 0:01:47.816301691, slope 8
0:02:33.946767140   449     0x12da0580 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:659:calculate_skew: delta -31698309, new min: -32239689
0:02:33.946772420   449     0x12da0580 DEBUG        rtpjitterbuffer rtpjitterbuffer.c:681:calculate_skew: skew -32751313, out 0:02:33.541312165

Queue full, dropping old packet seems to be present in the logs which would indicate packet loss at the source. What are potential causes for this issue (drop-on-latency=True in rtspsrc would be one) and why does it happen on increasing the number of streams? As suggested in the topic, I have already tried using drop-on-latency=False in the rtspsrc which leads me to the same issue as the author of that topic faced i.e the output stream starts lagging behind the input stream significantly over a period of time.

Is there any other solution to the video artifact issue without sacrificing the FPS by setting drop-frame-interval in the decoder? EDIT - I see the video artifacts are back when I increase the number of streams from 5 to 10 even with the decoder ‘drop-frame-interval’ property set which means this solution will not scale.

if only using one Dahua camera, will the app run well?

No, I see Queue full, dropping old packet in the logs when using a single Dahua RTSP source within a few mins of starting the application. I don’t see this log when using a single HIKVISION camera source. What might be the reason for it to differ across the cameras?

If I increase the number of RTSP streams with just HIKVISION input the packet drop logs appear and video artifacts are generated.

fanzh · June 15, 2024, 3:16am

RTSP receiving will use Gstreamer opensource plugin rtspsrc. Queue full, dropping old packet is the log of Gstreamer opensource code. it means there is RTSP receiving issue. since you did not provide pipeline, I don’t know how did you receive.
from your test, only using one Dahua camera, there is still receiving issue. here are some workarounds.

using Dahua camera configuration tool, lower the bitrate, fps. it will reduce the receiving issue at the client side.
packets loss will lead to artifacts. increasing latency and drop-on-latency=false will reduce latency and packets loss.
drop-on-latency=true will make the receiving queue don’t drop packets, but will increase lagging.

yingliu · July 9, 2024, 2:31am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · July 23, 2024, 2:31am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.