I’m running DeepStream based pipeline in production with RTSP streams. One of the issues I’m facing is, if one of the cameras go down, the entire pipeline stops. I’m using a dynamic engine so my model can handle runtime changes in number of sources. But how do I make it more fail safe? I want it to dynamically reduce the number of streams in case one goes down, and then put it back in when it’s online. Any recommendations?
• Hardware Platform (Jetson / GPU): Tesla T4
• DeepStream Version: 5
What does it mean “go down”?
We have samples for " Runtime source addition/removal with DeepStream", please refer to it.
Connect a CCTV camera to DeepStream and then power it off, that’s what I mean by going down. Power loss / network failures / anything bad that precludes the RTSP stream from reaching the inference server counts as going down. In this case, I’d like the other streams to continue running regardless, and once the stream comes back up, meaning, when the power is back / network is back / RTSP can connect through, then I’d want it to resume inference on it. This behaviour shouldn’t affect the performance of the other streams. I’ll check the link you shared, but would it achieve something similar that I’m looking for? Thanks.
gstreamer rtspsrc plugin is just a RTSP client implementation. It can not detect RTSP server errors such as power down,… You need to use other method to detect the error and remove the rtspsrc when error happens. We have sample codes for " Runtime source addition/removal with DeepStream"
Yeah I know, that’s why I am wondering if there’s a design pattern you have in place because I think it should be quite a prevalent problem with DeepStream installations with other customers. Right now that’s the only thing stopping me from deploying it to customers is even if one of the cameras go down, the entire pipeline is stopped. How have others solved this similar problem? Can you point me to any documentation that I can go through?
The link that you have shared is not helpful, it just adds and removes a source during runtime. I don’t want to do that. I want to handle a random camera going down and stalling the pipeline. Thanks in advance!
deepstream-app can continue to work even when one or several streams broken. The only case it stops working is that the client received “EOS” from server or error message from pipeline. In this case, application needs to finish the pipeline and re-start the pipeline. Application can decide what to do when there is error happens.