Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
GPU(dGPU) NVIDIA RTX 6000 Ada Generation
• DeepStream Version
Deepstream 7.1
base docker image: <nvcr.io/nvidia/deepstream:7.1-triton-multiarch>
• TensorRT Version
10.3.0
• NVIDIA GPU Driver Version (valid for GPU only)
Currently unavailable (field server) [To be checked]
• Issue Type( questions, new requirements, bugs)
Questions / Possible Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
The issue occurs intermittently during long-term (24/7) operation in the field environment, and we have not been able to reproduce it consistently in our development environment.
The application uses nvmultiurisrcbin (use-nvmultiurisrcbin=1) with 8 RTSP sources.
The issue has been observed after a temporary interruption of an RTSP stream, such as a network interruption, RTSP server failover, or temporary RTSP service interruption.
When the issue occurs, nvurisrcbin starts its internal reconnection procedure, but the affected source may enter a continuous reconnection loop:
No data from source since last 20 sec. Trying reconnection
Could not send message. (Received end-of-file)
Resetting source X, attempts: 1
Even after the RTSP stream becomes available again, the affected source does not resume streaming.
Other RTSP sources continue operating normally, and restarting only the DeepStream application immediately restores the affected source.
We have attempted to reproduce the same behavior in our development environment by intentionally interrupting and restoring RTSP streams, but so far the issue has not been reproduced consistently.
Therefore, we currently do not have deterministic reproduction steps. The detailed configuration, field observations, and logs are provided below.
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
N/A — This is not a new feature request.
1. Environment
- OS: Windows Server 2022 / WSL2 (Ubuntu 22.04)
- GPU: NVIDIA RTX 6000 Ada Generation
- DeepStream Version: 7.1
- TensorRT Version: 10.3.0
- Docker Image:
nvcr.io/nvidia/deepstream:7.1-triton-multiarch - GStreamer: Version bundled with DeepStream
- Source Plugin:
nvmultiurisrcbin(use-nvmultiurisrcbin=1)- We understand that each RTSP source is internally handled by
nvurisrcbin.
- We understand that each RTSP source is internally handled by
- RTSP Streams: 8 channels
- RTSP/RTP Transport: TCP (
select-rtp-protocol=4) - Muxer: Legacy
nvstreammux - Operating Mode: 24/7 continuous operation
Source List Configuration
[source-list]
use-nvmultiurisrcbin=1
use-source-list-file=1
source-list-file=../../../info/source_list.txt
stream-name-display=0
max-batch-size=26
http-ip=localhost
http-port=9000
sgie-batch-size=26
Source Configuration
[source-attr-all]
enable=1
type=3
num-sources=1
gpu-id=0
cudadec-memtype=0
select-rtp-protocol=4
latency=100
rtsp-reconnect-interval-sec=20
rtsp-reconnect-attempts=-1
drop-frame-interval=3
Streammux Configuration
[streammux]
gpu-id=0
live-source=0
batch-size=30
sync-inputs=0
batched-push-timeout=100000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0
drop-pipeline-eos=1
Additional Configuration Notes
Although only eight RTSP channels are currently in use, the system has been designed to support up to 26 channels.
Therefore:
max-batch-size=26is intentionally configured based on the target maximum number of channels.sgie-batch-size=26is configured accordingly to match the expected maximum batch size.streammux batch-size=30is intentionally configured with additional margin for future expansion.
We do not believe these settings are related to the observed issue because:
drop-frame-interval=3reduces the effective input frame rate.batched-push-timeout=100000provides sufficient batching time.- The pipeline operates normally for long periods before the reconnect issue occurs.
2. Problem Description
Occasionally, an RTSP stream is temporarily interrupted due to external events such as:
- Temporary network interruption
- RTSP server failover
- Temporary RTSP service interruption
When this occurs, nvurisrcbin automatically starts its internal reconnection procedure.
Typical logs observed in the field are:
No data from source since last 20 sec. Trying reconnection
Could not send message. (Received end-of-file)
Resetting source X, attempts: 1
The reconnection procedure continues indefinitely.
However, even after the RTSP stream becomes fully available again, the affected channel never resumes streaming.
The DeepStream application itself continues operating normally, and all other RTSP channels continue processing video correctly.
Only the affected channel remains disconnected until the entire DeepStream application is restarted.
No restart or configuration change is required on the RTSP server side. Restarting only the DeepStream application immediately restores the affected channel.
3. Additional Observations
- The issue occurs intermittently during long-term operation.
- It is difficult to reproduce in the development environment.
- We confirmed through the VMS client that the RTSP stream itself had already recovered.
- Other RTSP channels continue operating normally.
- Restarting the DeepStream application always restores the failed channel immediately.
- Reconnection logs continue indefinitely.
- The reconnect counter repeatedly prints:
Resetting source X, attempts: 1
Although reconnection is continuously attempted, the counter never increases and always remains at attempts: 1.
4. Technical Observations and Hypotheses
Based on the collected logs and observations from both MediaMTX and the field VMS environment, we are considering the following possibilities.
The items below are hypotheses based on observed behavior and may not reflect the actual internal implementation.
Hypothesis 1. Reconnection repeatedly fails during RTSP session initialization
It appears that the TCP connection may be established successfully, but the RTSP session initialization (DESCRIBE, SETUP, PLAY) or the subsequent RTP data reception does not complete successfully, causing the reconnection procedure to repeat.
We would also like to understand whether an existing RTSP session on the server side could remain active for some time or whether delayed session cleanup could affect creation of a new RTSP session.
Hypothesis 2. Effect of reconnect interval
The current configuration is:
rtsp-reconnect-interval-sec=20
We would like to understand whether this value only controls the no-data detection/reconnect timing, or whether it also affects any part of the RTSP session cleanup and reconnection sequence.
Would using a shorter reconnect interval (for example, 2–5 seconds) have any effect on recovery reliability, or would it only reduce the time before a reconnect is triggered?
Hypothesis 3. Internal reconnect behavior of nvurisrcbin
We would like to understand whether nvurisrcbin completely destroys and recreates the following objects or contexts during reconnect:
rtspsrc- TCP socket context
- RTSP session context
- Depayloader
- Decoder
- Source bin
Or whether it simply performs GStreamer state transitions while reusing some of the existing internal objects.
If internal contexts are reused, could an incompletely reset state cause the reconnect loop to continue indefinitely?
Hypothesis 4. Internal state retention in Legacy nvstreammux
Another possibility we are considering is the internal state maintained by Legacy nvstreammux.
Could nvstreammux retain source-specific information such as:
- EOS state
- Sink pad state
- Source activation state
- Other source-specific internal state
after repeated reconnect attempts?
If these internal states are not fully reset, could the RTSP connection and decoder recover successfully while the generated buffers are still unable to pass through nvstreammux to downstream elements?
Since restarting the entire DeepStream application immediately restores the stream, we would like to know whether the issue could be related to internal pipeline state, including nvstreammux.
5. Questions
1. Expected reconnect behavior and internal reconnect state machine
First, is this behavior expected for nvurisrcbin, or should nvurisrcbin normally be able to recover from temporary RTSP/network interruptions without requiring an application restart?
We would also like to understand the exact reconnect sequence and timing used by nvurisrcbin.
Our current understanding is:
Last video buffer received
↓
No buffer for 20 seconds
↓
"No data from source"
↓
Shutdown of the existing RTSP session
↓
New TCP / RTSP connection attempt
↓
DESCRIBE / SETUP / PLAY
↓
First RTP packet or decoded video buffer
↓
Reconnect again if unsuccessful
Could you please clarify the following?
- When exactly does the first reconnect begin after the last buffer is received?
- When is the existing RTSP session terminated?
- Does
nvurisrcbinsendPAUSEorTEARDOWN? - If so, how long is the internal timeout?
- When does the new TCP connection begin?
- If the RTSP connection succeeds but no video buffer is received, when is the next reconnect attempted?
- Is reconnect behavior different between TCP connection failures and RTSP session failures?
- At which stage is
Resetting source X, attempts:Nprinted? - Which event resets or increments the
attemptscounter? (TCP connect,DESCRIBE,SETUP,PLAY, first RTP packet, first decoded frame, etc.)
2. Internal element handling during reconnect
During reconnect:
- Is
rtspsrcdestroyed and recreated? - Are the depayloader and decoder also recreated?
- Is the entire source bin recreated?
- Or are existing elements reused through state transitions (
NULL → READY → PAUSED → PLAYING)?
Are the following contexts also recreated?
- TCP socket context
- RTSP connection context
- RTSP session context
- RTP-related context
3. Conditions that can lead to an infinite reconnect loop
Are there any known internal conditions under which reconnect continues indefinitely while the stream never recovers?
For example, could any of the following remain in an inconsistent state?
- Socket context
- RTSP session context
- Internal
rtspsrcstate - Source bin state
- Decoder state
4. Internal state of Legacy nvstreammux
Can Legacy nvstreammux retain source-specific internal state, such as EOS or sink pad state, after repeated reconnect attempts?
Could this prevent recovered buffers from reaching downstream elements even though the source has successfully resumed decoding?
Could live-source=0 affect source recovery or timestamp/buffer handling after an RTSP reconnection?
If this is possible, is there any recommended debug method or GST_DEBUG category that can help determine whether:
- the RTSP session has successfully recovered,
- decoded buffers are being generated by the source bin, and
- those buffers are being accepted and forwarded by
nvstreammux?
5. Recommended reconnect interval
The current configuration is:
rtsp-reconnect-interval-sec=20
rtsp-reconnect-attempts=-1
Does NVIDIA recommend a shorter reconnect interval for long-running RTSP deployments?
Could this parameter affect:
- Session cleanup
- Reconnect success rate
- Recovery time
6. Recommended runtime recovery method
What is NVIDIA’s recommended runtime recovery procedure for restoring only a failed RTSP source without restarting the entire DeepStream application?
For example:
- Recreating only
nvurisrcbin - Removing and re-adding the source pad
- Resetting
nvstreammux - Using the
nvmultiurisrcbinruntime/REST API
Are any of these considered best practice?
7. Known Issues
Are there any known bugs, patches, release notes, or best practices related to long-running RTSP reconnect behavior in DeepStream 7.1?
In particular, are there any known reconnect-related issues involving:
nvmultiurisrcbin- Internal
nvurisrcbin rtspsrc- Legacy
nvstreammux
We would greatly appreciate any explanation regarding the internal reconnect state machine and the recommended recovery strategy for long-running production environments.