FPS drops to 0.2 after some time in Deepstream 5.0 python app

I am implementing a Deepstream pipeline and I am not able to run my server for more than 10 to 15 mins. I think the issue is with multiple RTSP source cause my pipeline is working fine with video file as input.

Exact issue : FPS drops for each source from 13 to 0.2 after 10 to 20 mins of initializing the serve
parameters are set to default cause my implementation is a mixture of all the samples present in deepstream_python_app repository.

My pipeline -
uridecodebin -> streammux -> pgie -> tracker -> nvtiler -> nvdsconv -> nvosd -> post_nvdsconv -> capsfiter -> nvv4l25264enc -> rtph264pay -> udpsink

Model - YoloV3 cutom class
No of RTSP source - 20

• GPU - RTX 2080 ti
• DeepStream Version - 5.0
• NVIDIA GPU Driver Version - 440.64

Hi,
Could you check if the issue is also seen in video frame rendering?

uridecodebin -> streammux -> pgie -> tracker -> nvtiler -> nvdsconv -> nvosd -> nveglglessink sync=false

Also for RTSP sources, please make sure live-source=1 is set to nvstreammux.

Hi Dane

I have tried with nveglglessink issue remains same . Also let me correct my question frame rate doesn’t drops to zero it gets to 0.2 from 13(intial), server is still fuctioning but at very low speed.
(sync = 0 in sink and live-source =1 checked.)
Also i got some lead :

  1. My pipeline works fine with 12 cameras instead of 20.
  2. I have changed batched-push-timeout in streammux to 1/max_fps(13) from 4000000 and i am not getting any fps drop instead of that it eliminates some of the particular set of cameras (or set fps to 0/0.2) and rest of the camera fps increased from 13 to 19 fps.

Is it happening due to some of camera source producing feeds at different rate. like some at 20 fps and some at 25 ???
Do i need to do any kind of profilling. if yes which tool should i use ?

please let me know how to fix this issue. Thanks is advance.

Hi,
Please share the config file for reference.

Also do you observe the same with default ResNet? Would like to know if it is specific to Yolo model.

Sorry for the late response, issue is resolved by setting -

  1. batched-push-timeout = 1 / (max_fps) in streammux - this resolved the issue of fps drop now server stops inferencing for some the source and continues for others.
  2. interval = 4 in pgie config file - this reduced the load on GPU and it started supporting all 20 cameras.

Thanks

1 Like