Nvurisrcbin does not reconnect when running multiple RTSP streams

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU (GeForce RTX 3060)
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.5.3.1
• NVIDIA GPU Driver Version (valid for GPU only) 535.104.12
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I’m using deepstream python test 3 to run inference on RTSP streams. I was looking for the functionality where I could reconnect to a stream when it gets disrupted, without restarting the pipeline. In the following link (Stream Disconnects In Deepstream Pipeline), it was suggested to use nvurisrcbin and set the “rtsp-reconnect-interval” property.

So I replaced uridecodebin with nvurisrcbin in the python script and set the reconnect interval as 10. After that if I give 1 RTSP stream as input, disconnect its LAN cable and then reconnect it, the pipeline reconnects to the stream successfully after a certain time and resumes the processing of the stream as shown in following logs:

Resetting source rtsp:stream1

**PERF:  {'stream0': 0.0} 

Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5769): gst_rtspsrc_reconnect (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not receive any UDP packets for 5.0000 seconds, maybe your firewall is blocking it. Retrying using a tcp connection.

**PERF:  {'stream0': 0.0} 
**PERF:  {'stream0': 0.0} 
**PERF:  {'stream0': 0.0} 
**PERF:  {'stream0': 0.0} 
**PERF:  {'stream0': 0.0} 

Decodebin child added: h265parse1 
Decodebin child added: capsfilter1 
Decodebin child added: nvv4l2decoder1 

**PERF:  {'stream0': 24.2} 
**PERF:  {'stream0': 25.0} 
**PERF:  {'stream0': 25.0}

However, if I give 2 streams as input and remove the LAN cable for stream 1, then processing for not only stream 1 but also stream 2 stops. And when I reconnect the LAN cable for stream 1, the pipeline is not able to reconnect to either of the streams as shown in following logs:

**PERF:  {'stream0': 0.0, 'stream1': 0.2} 

Resetting source rtsp://stream1
Warning: gst-stream-error-quark: No data from source rtsp://stream1 since last 10 sec. Trying reconnection (1): gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin

**PERF:  {'stream0': 0.0, 'stream1': 0.4} 

Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5769): gst_rtspsrc_reconnect (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not receive any UDP packets for 5.0000 seconds, maybe your firewall is blocking it. Retrying using a tcp connection.

**PERF:  {'stream0': 0.0, 'stream1': 0.2} 

Decodebin child added: h265parse2 
Decodebin child added: capsfilter2 
Decodebin child added: nvv4l2decoder2 

Resetting source rtsp://stream2
Warning: gst-stream-error-quark: No data from source rtsp://stream2 since last 10 sec. Trying reconnection (1): gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-01/GstDsNvUriSrcBin:uri-decode-bin

**PERF:  {'stream0': 0.18, 'stream1': 0.43} 

Resetting source rtsp://stream1
Decodebin child added: h265parse3 
Decodebin child added: capsfilter3 
Decodebin child added: nvv4l2decoder3 

**PERF:  {'stream0': 1.2, 'stream1': 1.2} 
**PERF:  {'stream0': 0.2, 'stream1': 0.0} 
**PERF:  {'stream0': 0.2, 'stream1': 0.0} 

Resetting source rtsp://stream1
Resetting source rtsp://stream2
Warning: gst-stream-error-quark: No data from source rtsp://stream1 since last 10 sec. Trying reconnection (1): gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin

**PERF:  {'stream0': 0.26, 'stream1': 0.0} 
**PERF:  {'stream0': 0.2, 'stream1': 0.0} 
**PERF:  {'stream0': 0.2, 'stream1': 0.0}

I’m using the deepstream:6.3-triton-multiarch docker as my environment.

Any help regarding this would be greatly appreciated, thank you.

1 Like
3

We’ll check this problem and notify you promptly once we have a conclusion.

1 Like
5

Any update regarding this?

6

We are still analyzing this issue. Because there are no issues with the C version(opt\nvidia\deepstream\deepstream\sources\apps\sample_apps\deepstream-test3\deepstream_test3_app.c) of the code, it may be a problem with Python binding or Python demo code logic.

1 Like
7

Have you replicated this issue on your side?

Also, do you have any suggestions regarding how I can handle this stream disconnection use case another way? I’m kind of stuck at the moment due to this.

8

Yes. It takes some time to analyze and solve. It can be updated in next version.

Currently, you can consider using the C/C++ version. Or you can try using the nvmultiurisrcbin.

9

Thank you for your response. Unfortunately, C/C++ is not an option for me at the moment so I will try with nvmultiurisrcbin.

10

You can also refer to the topic 270729.

streammux.set_property('batched-push-timeout', 40000)
1 Like
11

Tested the suggested change with 2 streams, reconnect functionality with nvurisrcbin is working fine now. Thank you.

Having read the documentation for batched-push-timeout, I’m curious if this property’s value has to change depending on the number of streams. For example if I am processing 25 RTSP streams simultaneously, would I have to change its value accordingly?

12

No. You can set this value based on the duration of the frame interval or other appropriate value.

1 Like
13

I tested on 6 public RTSP streams, for around 40-45 mins the streams were running fine. But then after that I could see that all streams had been disrupted, which I confirmed from the logs and display. The reconnecting functionality tried to reconnect for quite some time but except for stream3, it was not able to do so as shown in the logs.

I checked the streams in both vlc media player and using a simple opencv script, they were running properly or at least had resumed running after the potential disruption. However, in the deepstream pipeline it did not resume the processing.

Something interesting I noticed in the logs is that for stream3, it was not being reset and also its FPS was showing in the range 0.8-1.2. Stream3 was the only stream which was reconnected to successfully.

Appreciate your help regarding this, thank you.

six-streams-rtsp-reconnection.txt (48.1 KB)

Note: I replaced the actual RTSP urls in the logs due to privacy.

14

Since they are public RTSP streams, could you attach the uri to us? And you ony make one change for our demo code, is that right?
Could you refer to the following method to build the rtsp serer and reproduce the issue?
https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/34

15

Unfortunately, the streams are from a client location so due to contractual and privacy reasons I can’t share them.

For code modification, I added nvurisrcbin and its properties (reconnect interval) plus instead of peoplenet, I’m using my own custom yolov5 model as primary-pgie.

Sure, I will go through the link you provided and try to reproduce the issue.