Deepstream app hangs after running couple of hours of operation

[Shannon_Govekar]

I’ve noticed after running my Deepstream app for a couple of hours, I get a WARNING from rtpjitterbuffer causing the app to pause and not recover.

0:20:10.046783588 56943 0x7f638801cc60 WARN         rtpjitterbuffer rtpjitterbuffer.c:572:calculate_skew: delta - skew: 0:00:36.560203239 too big, reset skew
0:20:10.059157217 56943 0x7f638801cc60 WARN         rtpjitterbuffer rtpjitterbuffer.c:572:calculate_skew: delta - skew: 0:00:01.016241288 too big, reset skew
0:20:10.071577556 56943 0x7f638801cc60 WARN         rtpjitterbuffer rtpjitterbuffer.c:572:calculate_skew: delta - skew: 0:00:01.017807461 too big, reset skew
0:20:10.087663136 56943 0x7f638801cc60 WARN         rtpjitterbuffer rtpjitterbuffer.c:572:calculate_skew: delta - skew: 0:00:28.534730510 too big, reset skew

My setup consists of 7 rtsp cameras running at 30hz with the following properties
Streammux → batch-size = 7; batched-pushed-timeout= 40000; buffer-pool-size=4;
Source → latency = 1000; udp-buffer-size = 2000000
Fakesink-> sync and sos both false

Any general idea as to what could be the issue? At the time of failure the number of frames in my batch reduce to 1 (instead of 7) and then just hangs.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.0.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 470.63
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

[Fiona.Chen]

@Shannon_Govekar

Any general idea as to what could be the issue?

No idea with just the WARN log.

For debugging RTSP related issue, you may enable the gstreamer log of rtspsrc. rtspsrc (gstreamer.freedesktop.org)
Basic tutorial 11: Debugging tools (gstreamer.freedesktop.org)

[KGerry]

Are you outputting to screen and have you been noticing delays in the rtsp output? In one of my ds-applications I can see ds output is lagging and eventually hanging (exceed max buffer?)

[Shannon_Govekar]

@Fiona.Chen I’m trying to replicate this issue which happens after a couple of hours (12 -24 or more) of operation.

My output is a fakesink. @KGerry you mean the source buffer or which buffer?

[Fiona.Chen]

Can you tell us how to reproduce the failure step by step?

[Shannon_Govekar]

Thanks Fiona. On further debugging, it seems like like it was rtsp server issue. one channel especially kept sending a teardown request causing the rtsp stream fail. Solved for the moment.

[arivarasan.e]

Hi @Shannon_Govekar, May I please know how you solved this? I have the same issue. Will be helpful if you can share your fix.

[Shannon_Govekar]

@arivarasan.e I used Wireshark and Gstreamer debug logs to see what’s the error when the application fails. We found that our rtsp stream had 3 channels, 2 video channels and 1 metadata channel, of which 1 video channel kept giving a TEARDOWN request. The fix is basically ignoring the corrupted channel. Let me know if that helps.