Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) GPU • DeepStream Version 5.0 • JetPack Version (valid for Jetson only) • TensorRT Version n/a • NVIDIA GPU Driver Version (valid for GPU only) 440.64
I’m using deepstream 5.0 python version to process multiple RTSP streams, and running into some issue.
The processing fps (calculated by get_fps in GETFPS class every 5 seconds) dropped to a very low value like 0.2 after a while. The more streams the source has, the shorter time the fps drops to 0.2. This could be due to my application need to save some frames into video clips during the callback function tiler_sink_pad_buffer_probe(), it may take too much time to interfere the pipeline.
The question is how it interfere the pipeline, what is the requirement for the callback execution time.
If I only keep a minimum processing in the callback function, take out the saving frames into video clips to other thread or asynchronous function, what is the best way to do so?
What is the usual way to handle such scenario when the callback function needs a lot of time to execute?
Thanks @Fiona.Chen, I’ve checked that link, it seems not the case I encountered.
I figured out some parameter matters. It’s batched-push-timeout. In Deepstream 4.0, the sample code set it to 40000 microseconds (40 ms), while in DS5.0, by default it’s 4000000 microseconds (4s). Per https://docs.nvidia.com/metropolis/deepstream/4.0/DeepStream_Plugin_Manual.pdf,
this parameter’s meaning is “Timeout in microseconds to wait after the first buffer is
available to push the batch even if a complete batch is not formed.”
With 4s setting, even I just did object detection without any frame processing, the fps could drop to 0.2. At that time, some streams were missing and the rest streams got 0.2 fps, I guess this could be it’s calculated every 5 seconds, 1 frame divide 5s is 0.2.
The question is: why the pipeline could not get frame from some streams? It’s supposed to wait for all frames from each stream within 4s, if after 4s, it still doesn’t get all frames, it will fill what it has to the pipeline. right? I’ve verified the rtsp streams were working well, it should get all frames from each stream quickly, not have to wait maximum 4s.
In contrast, after I changed the parameter batched-push-timeout to 40ms as it is in DS4.0, the problem is gone. Everything works fine. I don’t understand why this timeout setting could cause the issue, unless it’s not timeout, rather it’s the minimum time it has to wait to get all frames.
Can someone please elaborate the parameter batched-push-timeout and understand the above test results?
Interesting, “batched-push-timeout” turns out to be very crucial to my application, which is using RTSP stream, multiple sources. It’s based on the sample deepstream_imagedata-multistream.
Do you have any suggestion to this question “If one of the sources is dysfunctional, then the deepstream will stop. Is there a way to skip the dysfunctional source, and continue?”
“batched-push-timeout” is to limit the waiting time for the situation when nvstreammux can not get enough buffers from sources in a period of time. If the sources work smoothly and feed buffers to nvstreammux quick enough, "“batched-push-timeout” is of no use.
Deepstream is a SDK but not an application, what will happen when “one of the sources is dysfunctional” depends on what the problem is. For example, if it is RTSP connection broken by network problem, the plugins will take no action but just report some errors, it is the application who needs to take actions according to the errors. Another example, if it is some small network delay problems, maybe the source plugin itself has some re-connection mechanism to handle the problem.
Thanks @Fiona.Chen. It makes sense how to handle the dysfunctional source.
Regarding the parameter “batched-push-timeout”, my initial understanding is the same as you explained, but based on the experiments I had done, it’s not that case in Deepstream 5.0 python SDK.
It’s more like a minimum time to wait to fill the pipeline. In the Deepstream 5.0 sample it’s 4s (not 40ms as it is in Deepstream 4.0), we happened to use that, and noticed that when we were processing multiple RTSP sources, it always waited for 4s to call the callback, which caused the fps to be 0.2. If I changed it to 2s, the fps would be 0.5, 1s will get 1 fps and so on. So as long as 1/batched-push-timeout > 15, we will likely get a fps 15.
Would you have any interest to investigate how this parameter behaves in Deepstream 5.0 python SDK when processing RTSP streams?