System latency after Deepstream running for several hours with multiple cameras

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

• Hardware Platform: Jetson NX
• DeepStream Version: 5.0
• JetPack Version: 4.4
• TensorRT Version: 7.1.0

Hi Sir/Madam:

Currently, I am trying to connect two cameras to one edge device (NX). Camera resolution is 1080P, with 15 FPS.

Cameras field of view are toward one object. (environment is not very complex. Maximum like 10 objects are inside FOV).

Problem we see is: after Deepstream runs for 5-8 hours without system rebooting, one camera will show around 5s delay/latency to the other camera.

This is wired. From video local display, we see the video stream is fluently in most cases. Thus, I am confused where this delay coming from.

I am using Python bindings for development. All the software version are up-to-date.

Here follows the pipeline with settings:

uridecodebin (rtsp video source) [same as SDK example][ I did NOT set property uri_decode_bin.set_property(“buffer-duration”,1) , uri_decode_bin.set_property(“buffer-size”,1). Will it induce delay? ]->
streammux [‘live-source’ is set 1, ‘streammux_timeout’ is set 4000000]->
pgie [Here we use Resnet10, which is the default pre-trained model provided by Nvidia]->
tracker →
tiler →
nvvideoconvert →
filter [video/x-raw(memory:NVMM), format=RGBA] →
nvvideoconvert - >
nvosd →
tee → queue1 → rtsp output
→ queue2 → video file output
→ queue3 → local display → nvegltransform → nveglglessink [I did **NOT** set property sync as false. Will it induce delay?]

Also, is there any logs of where I may find out the root cause?

Thanks a lot for your help.

3

Neither nvmultistreamtilter nor nvstreamdemux is used in your pipeline? What will you see in the output?

4

Can you reproduce the problem with deepstream-test3?

5

I use nvmultistreamtiler (which is the tiler above), but I didn’t use nvstreamdemux in my current pipeline. Any suggestion?

6

Can you reproduce the problem with deepstream-test3?

7

Ok. Let me try.

8

One thing to remind you, the line of “streammux.set_property(‘batched-push-timeout’, 4000000)” in deepstream_test_3.py should be modified to “streammux.set_property(‘batched-push-timeout’, 40000)”.

9

For your case, 15FPS camera, the “batched-push-timeout” value should be 66666.

10

How this value is calculated?

11

1000000 us / FPS

12

Hi Fiona:

I have tested my codes with 2 cameras over 15 hours. Currently two camera does not have latency problems. I guess maybe value of “batched-push-timeout” helps. Also, I am not quite sure whether two cameras are with same input FPS, because one camera is set 15 fps. The other one should also set 15 fps. It may also be set 30 fps by default. Maybe the codes inside AI inferencing function is too heavy. I will continue investigating and give you feedback.