I’m using Deepstream 7.1 service-maker in C++, to create pipelines which consume RTSP streams from MediaMTX.
After a temporary outage where all upstream RTSP sources went down, my pipelines did not reconnect when the streams came back up. We rely on nvurisrcbin reconnection settings (which have worked for partial/individual feed drops before). In this scenario, pipelines did not recover/reconnect to MediaMTX despite the feeds being live again until we manually restarted the deepstream pipelines.
The snippet below shows my nvurisrcbin setup. I can share more of the pipeline code if needed.
for (uint i = 0; i < num_sources; i++)
{
std::string name = "src_" + std::to_string(i);
pipeline.add("nvurisrcbin", name, "uri", uris[i], "rtsp-reconnect-interval", 15, "rtsp-reconnect-attempts", -1);
}
Here are my mediamtx config file and logs from one of the pipelines. Please note I have replaced all rtsp urls and IP addresses with placeholders. ds_pipeline_logs.log (20.8 MB) mediamtx.txt (24.5 KB)
Before outage, did the app work well? were the filesink output of three RTSP sources fine?
nvurisrcbin is opensource. it leverages Gstreamer rtspsrc to input RTSP source. From the log ds_pipeline_logs.log, the app tried many times to reconnect three sources. All failures were caused by “Could not send message. (Received end-of-file) of rtspsrc”.
Could you use the following cmd to test src0 on the same machine running the Deepstream app, and share the running log? Thanks!
Yes, did not seem to have major issues before the outage. We have a rtspsink output for our pipelines, which produced rtsp outputs fine. We even had some stream outages before, which were handled by the nvurisrcbin reconnect properly.
Another pipeline today went down with the same issue, with similar logs to the one I sent before. Other pipelines were still ok, and have validated that the rtsp input was fine when the pipeline in question was stuck in a reconnect loop. We’re ideally trying to pinpoint why the reconnection fails randomly and what steps can we take to prevent this.
Attaching logs from the command, ran for 1min 30s after which I did a keyboard interrupt. Once again, ip_addresses and rtsp urls have been replaced with placeholders gst_log.txt (114.6 KB)
I tested nvurisrcbin RTSP reconnection on DS7.1 with “gst-launch-1.0 nvurisrcbin uri=rtsp://xxx rtsp-reconnect-interval=15 rtsp-reconnect-attempts=-1 ! fakesink”, but failed to reproduce this issue. could you use this simplified pipeline to test? and help to reproduce this issue？I killed mediamtx(v1.8.4)with “kill -9”. Here is the running log log-1002.txt (8.0 KB).
Sorry for the late reply.
I’ve already tried to recreate this issue multiple times on my local with a similar setup like yours, and was unable to reproduce the issue.
We had another outage last week, where all streams were down, and all pipelines were stuck in a reconnect loop. Unfortunately, restarting the pipelines did not fix the issue this time. Once again, I validated that the original input streams were still up and running by connecting to them.
What did fix the issue though, was this change in the mediamtx config:
# Transport protocol used to pull the stream. available values are "automatic", "udp", "multicast", "tcp".
rtspTransport: tcp
this was previously set to rtspTransport: automatic. Which makes me think if there was an issue reconnecting to streams using udp before?
I hope we do not encounter this issue again, but for the time being, I’m setting up a mechanism to restart the pipeline when it gets stuck in the reconnection loop. I tried reading the reconnection gstreamer messages from the pipeline class in our service-maker implementation, but did not have much luck with that.
Could I have some guidance around a solution where after a set number of reconnections, the pipeline is restarted? Currently when i set the nvurisrcbin property rtsp-reconnect-attempts to something like 5, after failing to reconnect for 5 attempts I see the pipeline state not changing at all.
In deepstream-app, if attempts is greater than threshold, the app will stop the source bin. Please refer to watch_source_status of /opt/nvidia/deepstream/deepstream/sources/apps/apps-common/src/deepstream_source_bin.c.
nvurisrcbin is opensource. In watch_source_status of /opt/nvidia/deepstream/deepstream-8.0/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp, the plugin will send “attempt-exceeded” to message bus, you can pass listener function to listen the message, then restart pipeline in the listener().
if one stream failed to reconnect, The common practice is only stopping the srcbin if the attempts is greater than threshold. not restart the pipeline because maybe some other streams already reconnect successfully. please refer to nvmultiurisrcbin, which uses nvurisrcbin and implements stopping srcbin mechanism.
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.