Handling Camera Disconnections in DeepStream without Freezing the Entire Application

[jovan.babic.94]

Hi everyone,

I’m currently developing a Deep Stream application aimed at counting vehicles at an intersection, utilizing input from 15 cameras. The input can be either RTPS or .mkv files, though the preference is for RTPS streams.

I’ve encountered a significant issue where if one or more camera feeds stop (due to receiving an EOS event or any other disruption), it causes the entire application to freeze. To address this, I implemented event handlers designed to restart streams when such disruptions occur. However, this solution introduces temporary freezes in the application until the affected streams are back up.

Originally, my application was structured around a single pipeline. Considering the possibility that this architecture might contribute to the problem, I experimented with multi-threading and subsequently attempted to create a main pipeline with 15 sub-pipelines. The main pipeline was intended to share elements like the streammux, tiler, and sink across the sub-pipelines. Unfortunately, I struggled with linking the elements properly.

Another layer of complexity comes from the use of the _AppCtx struct as the main context for managing elements, which I’ve found somewhat challenging.

My goal is to ensure that when one or more camera feeds are disrupted, only the affected feeds freeze and are restarted, without impacting the continuity of the entire application. I believe this is achievable, but given my relative inexperience with Deep Stream, I might be overlooking some aspect of its capabilities or best practices.

Has anyone here dealt with similar challenges or can offer insights into how to architect this solution more effectively? Specifically, how can I manage multiple camera feeds in such a way that issues with individual feeds do not disrupt the entire application? Any advice on handling element linking in a multi-pipeline setup or managing the application context more efficiently would also be greatly appreciated.

Thank you in advance for your guidance and suggestions!

[yuweiw]

Could you reproduce your issues in one of our demos? Baiscally, you need to monitor the bus messages of each video source and then do your own logical control there.

[jovan.babic.94]

Thank you for your suggestion. I have indeed based my application on the deepstream-nvdsanalytics-test example and made some modifications to suit my needs. My application is designed to handle 15 streams simultaneously.

My primary goal is to ensure the application’s robustness in handling stream disruptions. Specifically, if a stream becomes corrupted, stops, or is unreachable, I want the application to continue processing the remaining streams without interruption. In other words, issues with one stream should not freeze or halt the processing of the other active streams.

I understand from your response that I need to monitor the bus messages of each video source and implement logical control accordingly. I will explore this approach further.

However, I would like to confirm: is it feasible to achieve continuous processing of the remaining streams in the event of disruptions with the DeepStream framework using the approach you suggested? If you have additional insights or resources that could assist me in achieving this, I would greatly appreciate your sharing them.

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

The usual way to do this is like in deepstream-app: add the reconnect logic. You can refer to our source code watch_source_status in the deepstream_source_bin.c file.