We follow a different approach, not using DeepStream app but rather a GStreamer pipeline that uses DeepStream elements and is launched with GstD. You can check the following wiki page for some DeepStream pipeline examples with gst-launch and GstD:
Using GstInterpipes, GstD and a python client we can recover from errors on the RTSP streams without stopping the inference pipeline. The diagram of the solution looks something like this:
When an RTSP stream stops receiving buffers or fails, we replace it with a dummy stream that sends videotestsrc buffers, so that the DeepStream pipeline doesnāt fail.
In the bus callback, you can get the āsrcā property of the message to get the source of the message, and if itās your rtsp source, you can handle the case differently, rather than quitting. I think then youll have to set the pipeline to a paused state, wait for that to complete, unlink the source, remove it from the pipeline, swap it out with another, link it, and restart the pipeline. Thatās my plan anyway since Iām working on the same problem. My RTSP sources are flaky AF, which is actually great to test with. There may be a way to do it without pausing the pipeline but I canāt find one. There isnāt much information on this out there and those who have solved this issue arenāt likely to share it. Any advice from gstreamer experts would be welcome.
@miguel.taylor, for the time being I would like to stick to the default deepstream pipeline.
@mdegans, I was trying a similar approach as to what you mentioned, but I have been unsuccessful till now. Kindly let me know, if you were able to make any progress. Any help is greatly appreciated.
I have also tried flushing up the pipeline and then restarting it all over again whenever it goes to EOS. I am no gstreamer expert and have very little hands-on experience with it. I am clueless as to why I am unable to restart the pipeline and make it to work properly.
I will let you know what if I figure out a solution, neophyte1. Gstreamer has a steep learning curve but the tutorials help. Unfortunately thereās a lot you just have to figure out as well. If you havenāt already found the debug stuff, this page is very useful. You can also hook up your option parser to gstreamerās so you can use --gst-debug-level at the command prompt. Frankly the .dot file has been a lifesaver for me. The ability to get a visual overview of your pipeline so you can see āaha. thatās not connectedā is priceless.
Hi,
Attach a patch that demonstrates reconnection in deepstream-app.
To enable reconnection, set source type to 4 and set rtsp-reconnect-interval-sec to the desired value. If no data is received within this set time, the app will force reconnection with the rtsp source.
FYR. reconnection.zip (4.47 KB)
Iāve the same problem. After 3-4 hours, deepstream app with RTSP source crashes getting this error:
WARNING from src_elem0: Could not read from resource.
Debug info: gstrtspsrc.c(5293): gst_rtspsrc_loop_udp (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRTSPSrc:src_elem0:
Unhandled return value -7.
ERROR from src_elem0: Could not read from resource.
Debug info: gstrtspsrc.c(5361): gst_rtspsrc_loop_udp (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRTSPSrc:src_elem0:
Could not receive message. (System error)
ERROR from src_elem0: Internal data stream error.
Debug info: gstrtspsrc.c(5653): gst_rtspsrc_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRTSPSrc:src_elem0:
streaming stopped, reason error (-5)
** INFO: <bus_callback:212>: Received EOS. Exiting ...
Reset source pipeline reset_source_pipeline 0x7f5d062080
,Quitting
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
ERROR from src_elem0: Could not write to resource.
Debug info: gstrtspsrc.c(5997): gst_rtspsrc_try_send (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRTSPSrc:src_elem0:
Could not send message. (System error)
ERROR from src_elem0: Could not write to resource.
Debug info: gstrtspsrc.c(8244): gst_rtspsrc_pause (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRTSPSrc:src_elem0:
Could not send message. (System error)
App run successful
Hi,
I applied the reconnection patch. I tried to run for 2 streams. After some time, I disconnected the network for a couple of minutes and then reconnected it. The pipeline resumed, but now the output from one of the streams became quite jittery and I began seeing this error on the terminal. -
Hi,
We have team trying to reproduce the failure. Will update once there is progress.
The patch is verified before public post. Probably there is core cases not considered.
Hi Danell,
For reproducing the isse, I would suggest to run deepstream reference app with the patch with 2-4 rtdp streams, and then disconnect the streams ( either switching off the network or something else). Donāt reconnect all the streams together at one time, but randomly.Kindly let me know if the issue can be resolved.
As far as adding and removing sources at runtime, Nvidia has this example here:
You should be able to port that logic to any language supported by GStreamer if GObject C isnāt your thing. Unfortunately there seems to be no way to change batch-size while elements are in the playing state, and Iām not sure how much testing has gone into changing it in other states. You may find it more reliable to discard the entire pipeline or restart the process. Some people use Python to manage a GStreamer subprocess for this purpose.
