DeepStream 7.1 nvurisrcbin fails to recover from RTSP disconnection and enters infinite reconnection loop

[heeyul.lee]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

GPU(dGPU) NVIDIA RTX 6000 Ada Generation

• DeepStream Version

Deepstream 7.1

base docker image: <nvcr.io/nvidia/deepstream:7.1-triton-multiarch>

• TensorRT Version

10.3.0

• NVIDIA GPU Driver Version (valid for GPU only)

Currently unavailable (field server) [To be checked]

• Issue Type( questions, new requirements, bugs)

Questions / Possible Bug

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

The issue occurs intermittently during long-term (24/7) operation in the field environment, and we have not been able to reproduce it consistently in our development environment.

The application uses nvmultiurisrcbin (use-nvmultiurisrcbin=1) with 8 RTSP sources.

The issue has been observed after a temporary interruption of an RTSP stream, such as a network interruption, RTSP server failover, or temporary RTSP service interruption.

When the issue occurs, nvurisrcbin starts its internal reconnection procedure, but the affected source may enter a continuous reconnection loop:

No data from source since last 20 sec. Trying reconnection
Could not send message. (Received end-of-file)
Resetting source X, attempts: 1

Even after the RTSP stream becomes available again, the affected source does not resume streaming.

Other RTSP sources continue operating normally, and restarting only the DeepStream application immediately restores the affected source.

We have attempted to reproduce the same behavior in our development environment by intentionally interrupting and restoring RTSP streams, but so far the issue has not been reproduced consistently.

Therefore, we currently do not have deterministic reproduction steps. The detailed configuration, field observations, and logs are provided below.

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

N/A — This is not a new feature request.

1. Environment

• OS: Windows Server 2022 / WSL2 (Ubuntu 22.04)
• GPU: NVIDIA RTX 6000 Ada Generation
• DeepStream Version: 7.1
• TensorRT Version: 10.3.0
• Docker Image: nvcr.io/nvidia/deepstream:7.1-triton-multiarch
• GStreamer: Version bundled with DeepStream
• Source Plugin: nvmultiurisrcbin (use-nvmultiurisrcbin=1)
  • We understand that each RTSP source is internally handled by nvurisrcbin.
• RTSP Streams: 8 channels
• RTSP/RTP Transport: TCP (select-rtp-protocol=4)
• Muxer: Legacy nvstreammux
• Operating Mode: 24/7 continuous operation

Source List Configuration

[source-list]
use-nvmultiurisrcbin=1
use-source-list-file=1
source-list-file=../../../info/source_list.txt
stream-name-display=0

max-batch-size=26
http-ip=localhost
http-port=9000
sgie-batch-size=26

Source Configuration

[source-attr-all]
enable=1
type=3
num-sources=1
gpu-id=0
cudadec-memtype=0
select-rtp-protocol=4
latency=100
rtsp-reconnect-interval-sec=20
rtsp-reconnect-attempts=-1
drop-frame-interval=3

Streammux Configuration

[streammux]
gpu-id=0
live-source=0
batch-size=30
sync-inputs=0
batched-push-timeout=100000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0
drop-pipeline-eos=1

Additional Configuration Notes

Although only eight RTSP channels are currently in use, the system has been designed to support up to 26 channels.

Therefore:

• max-batch-size=26 is intentionally configured based on the target maximum number of channels.
• sgie-batch-size=26 is configured accordingly to match the expected maximum batch size.
• streammux batch-size=30 is intentionally configured with additional margin for future expansion.

We do not believe these settings are related to the observed issue because:

• drop-frame-interval=3 reduces the effective input frame rate.
• batched-push-timeout=100000 provides sufficient batching time.
• The pipeline operates normally for long periods before the reconnect issue occurs.

2. Problem Description

Occasionally, an RTSP stream is temporarily interrupted due to external events such as:

• Temporary network interruption
• RTSP server failover
• Temporary RTSP service interruption

When this occurs, nvurisrcbin automatically starts its internal reconnection procedure.

Typical logs observed in the field are:

No data from source since last 20 sec. Trying reconnection
Could not send message. (Received end-of-file)
Resetting source X, attempts: 1

The reconnection procedure continues indefinitely.

However, even after the RTSP stream becomes fully available again, the affected channel never resumes streaming.

The DeepStream application itself continues operating normally, and all other RTSP channels continue processing video correctly.

Only the affected channel remains disconnected until the entire DeepStream application is restarted.

No restart or configuration change is required on the RTSP server side. Restarting only the DeepStream application immediately restores the affected channel.

3. Additional Observations

• The issue occurs intermittently during long-term operation.
• It is difficult to reproduce in the development environment.
• We confirmed through the VMS client that the RTSP stream itself had already recovered.
• Other RTSP channels continue operating normally.
• Restarting the DeepStream application always restores the failed channel immediately.
• Reconnection logs continue indefinitely.
• The reconnect counter repeatedly prints:

Resetting source X, attempts: 1

Although reconnection is continuously attempted, the counter never increases and always remains at attempts: 1.

4. Technical Observations and Hypotheses

Based on the collected logs and observations from both MediaMTX and the field VMS environment, we are considering the following possibilities.

The items below are hypotheses based on observed behavior and may not reflect the actual internal implementation.

Hypothesis 1. Reconnection repeatedly fails during RTSP session initialization

It appears that the TCP connection may be established successfully, but the RTSP session initialization (DESCRIBE, SETUP, PLAY) or the subsequent RTP data reception does not complete successfully, causing the reconnection procedure to repeat.

We would also like to understand whether an existing RTSP session on the server side could remain active for some time or whether delayed session cleanup could affect creation of a new RTSP session.

Hypothesis 2. Effect of reconnect interval

The current configuration is:

rtsp-reconnect-interval-sec=20

We would like to understand whether this value only controls the no-data detection/reconnect timing, or whether it also affects any part of the RTSP session cleanup and reconnection sequence.

Would using a shorter reconnect interval (for example, 2–5 seconds) have any effect on recovery reliability, or would it only reduce the time before a reconnect is triggered?

Hypothesis 3. Internal reconnect behavior of nvurisrcbin

We would like to understand whether nvurisrcbin completely destroys and recreates the following objects or contexts during reconnect:

• rtspsrc
• TCP socket context
• RTSP session context
• Depayloader
• Decoder
• Source bin

Or whether it simply performs GStreamer state transitions while reusing some of the existing internal objects.

If internal contexts are reused, could an incompletely reset state cause the reconnect loop to continue indefinitely?

Hypothesis 4. Internal state retention in Legacy nvstreammux

Another possibility we are considering is the internal state maintained by Legacy nvstreammux.

Could nvstreammux retain source-specific information such as:

• EOS state
• Sink pad state
• Source activation state
• Other source-specific internal state

after repeated reconnect attempts?

If these internal states are not fully reset, could the RTSP connection and decoder recover successfully while the generated buffers are still unable to pass through nvstreammux to downstream elements?

Since restarting the entire DeepStream application immediately restores the stream, we would like to know whether the issue could be related to internal pipeline state, including nvstreammux.

5. Questions

1. Expected reconnect behavior and internal reconnect state machine

First, is this behavior expected for nvurisrcbin, or should nvurisrcbin normally be able to recover from temporary RTSP/network interruptions without requiring an application restart?

We would also like to understand the exact reconnect sequence and timing used by nvurisrcbin.

Our current understanding is:

Last video buffer received
        ↓
No buffer for 20 seconds
        ↓
"No data from source"
        ↓
Shutdown of the existing RTSP session
        ↓
New TCP / RTSP connection attempt
        ↓
DESCRIBE / SETUP / PLAY
        ↓
First RTP packet or decoded video buffer
        ↓
Reconnect again if unsuccessful

Could you please clarify the following?

• When exactly does the first reconnect begin after the last buffer is received?
• When is the existing RTSP session terminated?
• Does nvurisrcbin send PAUSE or TEARDOWN?
• If so, how long is the internal timeout?
• When does the new TCP connection begin?
• If the RTSP connection succeeds but no video buffer is received, when is the next reconnect attempted?
• Is reconnect behavior different between TCP connection failures and RTSP session failures?
• At which stage is Resetting source X, attempts:N printed?
• Which event resets or increments the attempts counter? (TCP connect, DESCRIBE, SETUP, PLAY, first RTP packet, first decoded frame, etc.)

2. Internal element handling during reconnect

During reconnect:

• Is rtspsrc destroyed and recreated?
• Are the depayloader and decoder also recreated?
• Is the entire source bin recreated?
• Or are existing elements reused through state transitions (NULL → READY → PAUSED → PLAYING)?

Are the following contexts also recreated?

• TCP socket context
• RTSP connection context
• RTSP session context
• RTP-related context

3. Conditions that can lead to an infinite reconnect loop

Are there any known internal conditions under which reconnect continues indefinitely while the stream never recovers?

For example, could any of the following remain in an inconsistent state?

• Socket context
• RTSP session context
• Internal rtspsrc state
• Source bin state
• Decoder state

4. Internal state of Legacy nvstreammux

Can Legacy nvstreammux retain source-specific internal state, such as EOS or sink pad state, after repeated reconnect attempts?

Could this prevent recovered buffers from reaching downstream elements even though the source has successfully resumed decoding?

Could live-source=0 affect source recovery or timestamp/buffer handling after an RTSP reconnection?

If this is possible, is there any recommended debug method or GST_DEBUG category that can help determine whether:

1. the RTSP session has successfully recovered,
2. decoded buffers are being generated by the source bin, and
3. those buffers are being accepted and forwarded by nvstreammux?

5. Recommended reconnect interval

The current configuration is:

rtsp-reconnect-interval-sec=20
rtsp-reconnect-attempts=-1

Does NVIDIA recommend a shorter reconnect interval for long-running RTSP deployments?

Could this parameter affect:

• Session cleanup
• Reconnect success rate
• Recovery time

6. Recommended runtime recovery method

What is NVIDIA’s recommended runtime recovery procedure for restoring only a failed RTSP source without restarting the entire DeepStream application?

For example:

• Recreating only nvurisrcbin
• Removing and re-adding the source pad
• Resetting nvstreammux
• Using the nvmultiurisrcbin runtime/REST API

Are any of these considered best practice?

7. Known Issues

Are there any known bugs, patches, release notes, or best practices related to long-running RTSP reconnect behavior in DeepStream 7.1?

In particular, are there any known reconnect-related issues involving:

• nvmultiurisrcbin
• Internal nvurisrcbin
• rtspsrc
• Legacy nvstreammux

We would greatly appreciate any explanation regarding the internal reconnect state machine and the recommended recovery strategy for long-running production environments.

[yuweiw]

@heeyul.lee

First, is this behavior expected for nvurisrcbin, or should nvurisrcbin normally be able to recover from temporary RTSP/network interruptions without requiring an application restart?

No. If there is indeed video data, the reconnection should have been successful.

@heeyul.lee

• When exactly does the first reconnect begin after the last buffer is received?

You can refer to our source code gstdsnvurisrcbin.cpp. The watchdog runs once per second. With an interval of 20 seconds, the first reconnect normally starts about 20–21 seconds after the last depay buffer.

@heeyul.lee

• When is the existing RTSP session terminated?

gst_element_set_state (GST_ELEMENT (src_bin->src_elem), GST_STATE_NULL) == GST_STATE_CHANGE_FAILURE

@heeyul.lee

• Does nvurisrcbin send PAUSE or TEARDOWN?

Any RTSP TEARDOWN, connection shutdown, or socket cleanup is performed internally by GStreamer rtspsrc as part of its transition to NULL. It’s not in the nvurisrcbin.

@heeyul.lee

• When does the new TCP connection begin?

[DeepStream/src/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp at main · NVIDIA/DeepStream · GitHub](https://gst_element_set_state (GST_ELEMENT (src_bin->src_elem), GST_STATE_PLAYING))

@heeyul.lee

• If the RTSP connection succeeds but no video buffer is received, when is the next reconnect attempted?

With an interval of 20 seconds you set

@heeyul.lee

• Is reconnect behavior different between TCP connection failures and RTSP session failures?

The watchdog does not distinguish between:

• TCP connection failure;
• RTSP DESCRIBE failure;
• RTSP SETUP failure;
• RTSP PLAY failure;
• server returning EOF;
• connected session with no RTP;
• RTP arriving but no complete depayloaded buffer.

All these cases eventually appear as “no depay buffer,” and the same reset_source_pipeline() path is used.

@heeyul.lee

• At which stage is Resetting source X, attempts:N printed?

https://github.com/NVIDIA/DeepStream/blob/main/src/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp#L1889

@heeyul.lee

• Which event resets or increments the attempts counter? (TCP connect, DESCRIBE, SETUP, PLAY, first RTP packet, first decoded frame, etc.)

https://github.com/NVIDIA/DeepStream/blob/main/src/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp#L1830

https://github.com/NVIDIA/DeepStream/blob/main/src/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp#L2272

@heeyul.lee

• Is rtspsrc destroyed and recreated?
• Are the depayloader and decoder also recreated?
• Is the entire source bin recreated?
• Or are existing elements reused through state transitions (NULL → READY → PAUSED → PLAYING)?

No. The source pipeline is not destroyed and recreated. Existing plugins are reused.

@heeyul.lee

Internal state of Legacy nvstreammux

Please set live-source=1 for your scenario.

Our suggestion:

1. Run in the following way and obtain the log.

GST_DEBUG="nvurisrcbin:6,nvmultiurisrcbin:6,rtspsrc:6,rtpjitterbuffer:5,rtph264depay:6,h264parse:5,decodebin:5,nvv4l2decoder:5,nvstreammux:6"

This will show whether recovery stops at RTSP, depay, parser, decoder, or mux.

2. You can try to dynamically add or remove the rtsp source as a workaround. If the source fails to connect for a long time, you can delete the source dynamically and then add it again.

3. Would you consider upgrading the DeepStream? Because all our subsequent maintenance is based on the latest version.

[heeyul.lee]

Thank you for the detailed explanation and for pointing us to the relevant locations in the source code.

Based on the code references you provided, we further reviewed the related logic in gstdsnvurisrcbin.cpp, including watch_source_status(), watch_source_async_state_change(), reset_source_pipeline(), rtspsrc_monitor_probe_func(), and the reconnect counter handling. We also performed additional reconnection tests in our own test environment.

During this analysis, we identified an important difference in the reconnect counter behavior between a normal persistent connection failure and the issue we observed at the field site.

1. Normal persistent connection failure

For example, when we intentionally made the RTSP connection continuously fail by turning off a camera, the reconnect attempts increased as expected:

Resetting source N, attempts: 1
Resetting source N, attempts: 2
Resetting source N, attempts: 3
...

In this case, the source does not successfully recover and no buffer is received, so the reconnect counter is not reset.

As a result, num_rtsp_reconnects continues to increase, and once the configured rtsp_reconnect_attempts limit is exceeded, we confirmed in our test that the source removal path in gstdsnvmultiurisrcbin.cpp is eventually reached.

2. Problem observed at the field site

In contrast, when the issue occurred at the field site, we continuously observed the following pattern:

Resetting source N, attempts: 1
Resetting source N, attempts: 1
Resetting source N, attempts: 1
...

While reviewing the reconnect logic based on the source code you referenced, we checked where num_rtsp_reconnects is incremented and reset.

We found that the reconnect attempt counter is incremented immediately before calling reset_source_pipeline(), in both the reconfiguring == TRUE and reconfiguring == FALSE paths.

Excluding the initial value set when the object is created, we identified two main conditions during reconnection that reset num_rtsp_reconnects:

1. When the source reaches the PLAYING state in watch_source_async_state_change()

2. When at least one buffer is observed on the depayloader src pad in rtspsrc_monitor_probe_func()

Therefore, based on the code flow, we understand that the field pattern

attempts: 1 → 1 → 1 → 1 → ...

indicates that num_rtsp_reconnects was being reset to 0 between reconnect cycles.

In other words, the field issue appears to have been different from a simple persistent RTSP connection failure.

Our current interpretation is that the reconnection progressed far enough for either the source to reach PLAYING or at least one depay buffer to be observed, which reset the reconnect counter. After that, buffer delivery stopped again, the watchdog detected the lack of depay buffers, and another reconnect cycle started.

As a result, each new reconnect cycle started again with attempts: 1. Therefore, the configured rtsp_reconnect_attempts limit was never reached, and the automatic source removal path in gstdsnvmultiurisrcbin.cpp was not triggered.

The difference we observed can be summarized as follows:

Item	Normal persistent connection failure	Field issue
Reconnection behavior	Connection/recovery continues to fail	Appears to progress to a certain stage and then stops again
PLAYING / depay buffer	Counter reset condition is not reached	Either PLAYING or buffer reception appears to occur
Reconnect counter reset	No	Occurs between reconnect cycles
num_rtsp_reconnects	1 → 2 → 3 → 4 → …	1 → 1 → 1 → 1 → …
rtsp_reconnect_attempts limit	Eventually reached	Never reached
Source removal	Eventually triggered	Not triggered
Result	Source is removed	Reconnection continues repeatedly

This analysis helped us understand why attempts: 1 continued to repeat and why the reconnect attempt limit was never reached.

Unfortunately, the field evaluation has now been completed. As part of closing the field evaluation, the test server was removed from the site and has also been formatted. Therefore, we no longer have access to the actual environment where the issue occurred, and it is no longer possible to collect additional logs or perform further reproduction and verification using the settings you suggested.

Also, we have not been able to reproduce the same attempts: 1 repeating behavior in our local test environment.

For these reasons, we were unable to determine whether reaching PLAYING or receiving a depay buffer was the actual condition repeatedly resetting the reconnect counter at the field site. More importantly, we could not determine the original root cause of why the reconnection progressed far enough to reset the counter but then failed to recover into stable, continuous streaming.

We believe the detailed GST_DEBUG configuration you suggested would have been very helpful in determining whether the recovery was stopping at the RTSP, depay, parser, decoder, or mux stage if we had been able to apply it to the affected field environment.

Unfortunately, given that the original field environment is no longer available, we will conclude our investigation of this issue for now.

If we encounter the same behavior again in a future deployment, we will use the GST_DEBUG settings you provided together with what we learned from this code analysis to collect more detailed information about the RTSP reconnect state transitions, depay buffer reception, and the exact timing and condition under which the reconnect counter is reset.

We will also consider the dynamic source remove/add approach you suggested as a workaround when a source fails to recover for an extended period, and we will consider upgrading to a newer DeepStream version for future deployments.

Thank you again for your detailed explanations and support.