Nvurisrcbin reconnect logic gets permanently stuck after a brief RTSP network blip (one camera "freezes" for good)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Environment

  • Platform: Jetson AGX Thor
  • JetPack: 7.0 (b128)
  • L4T: R38.2.0
  • DeepStream SDK: 8.0.0
  • GStreamer: 1.24.2
  • TensorRT: 10.13.3.9 / CUDA 13.0
  • Pipeline: Python, pyservicemaker/GStreamer bindings

Setup
Two IP cameras (entry, exit) over RTSP, each fed into the pipeline through its own nvurisrcbin → shared nvstreammux → nvinfer (batch=2). Relevant nvurisrcbin properties:

  • select-rtp-protocol = 4 (force TCP)
  • rtsp-reconnect-interval set (few seconds)
  • Stream: 2688x1520 @ 20 fps, H.264 (confirmed via ffprobe direct connection — camera and network are healthy)

Symptom
Only the exit camera’s stream intermittently stops delivering frames entirely — permanently, not a brief drop. Entry keeps running fine, sometimes even during the same window. Once it happens, it does not recover on its own; frame counters for that source freeze forever until we intervene.

Observed root cause
A short (~3s), completely ordinary network blip triggers nvurisrcbin’s internal watchdog to attempt a reconnect. During that reconnect, its own cleanup step (gst_rtspsrc_pause / gst_rtspsrc_close) fails against the already-broken socket. Because the old RTSP session is never torn down successfully, a new one is never established either — nvurisrcbin is left retrying try_send/pause/close against a dead socket indefinitely, roughly every 4–8s, forever.

Log signature (GST_DEBUG / journal):
GST WARN: gst-stream-error-quark: No data from source since last 3 sec. Trying reconnection (1)
[app] RTSP stream down, reconnecting
[app] RTSP stream recovered ← misleading: one frame slipped through, not real recovery
GST WARN: Could not write to resource. gst_rtspsrc_try_send
GST WARN: Could not write to resource. gst_rtspsrc_pause ← cleanup itself fails
GST WARN: Could not read from resource. gst_rtspsrc_try_send
GST WARN: Could not write to resource. gst_rtspsrc_close ← close also fails
[repeats try_send/pause/close failures every ~4-8s, indefinitely]
Sometimes preceded by the DeepStream-native line: Resetting source -1, attempts: N.

Ruled out

  • Camera/network health — verified via ping (0% loss, <1ms) and a direct ffprobe RTSP connection (instant, clean 2688x1520@20fps) while the production stream was frozen.

  • Hardware decoder (nvv4l2decoder) — the freeze is entirely at the RTSP session layer, before decode is ever reached.

  • GPU/resource contention from other processes on the box.

  • One-off fluke — reproduced 4–5 times in a single day, recurrence intervals ranging from ~9 minutes down to under a minute after a fresh restart.

    Note : I have checked with a new camera and with a new lan cable yesterday in the galaxy store it is the same results with the Hardware decoders,
    but when I switched back to the CPU decoders and encoders I was not facing a issue .

Question
Is there a known fix, patch, or nvurisrcbin/rtspsrc property (e.g. something in smart-reconfig, latency, or an internal timeout) that avoids the reconnect getting wedged in the first place? Is this a known issue in DS 8.0 on Thor/JetPack 7.0, and is there a tracking bug number we can watch?

3

I have checked with a new camera and with a new lan cable yesterday in the galaxy store it is the same results with the Hardware decoders,
but when I switched back to the CPU decoders and encoders I was not facing a issue .

4
  1. Can the two RTSP streams be played correctly before the “reconnect” being triggered?
  2. From the description, you set “rtsp-reconnect-interval=3” and “select-rtp-protocol=RTP_PROTOCOL_TCP”, what is your configuration with “drop-on-latency” property?

Is the ffprobe command/app use exactly the same configuration as you set with nvurisrcbin? Is the transfer protocol “tcp” too?
4.

Do you mean there is no reconnect attempt being triggered when try with software “decoders” and “encoders”? From your previous description, I can’t find where and how you use “encoder” in your pipeline. Can you elaborate the complete pipeline or post the pipeline graph?
5. Do you know your camera’s “IDR interval” in the video settings?

5

Thanks Fiona . Answers below:

  1. Do both streams play fine before the reconnect?

Yes. Both cameras start fine and give frames. The freeze only happens after a short network blip starts nvurisrcbin’s reconnect — never at startup. The camera itself stays healthy the whole time (ping works, and ffprobe pulls a clean 2688x1520@20fps over TCP even during the freeze). Right now one camera’s source has been frozen in the pipeline for ~4 hours, while the other camera on the same pipeline keeps running at ~20 fps.

  1. drop-on-latency setting?

Not set — it is at default. On nvurisrcbin we only set:

  • select-rtp-protocol = 4 (TCP)
  • rtsp-reconnect-interval = 3

We don’t set latency either. Happy to try a value if you suggest one.

  1. Does ffprobe use the same TCP transport?

Yes — ffprobe uses -rtsp_transport tcp, same as select-rtp-protocol=4. Even while a stream is frozen, ffprobe connects to the same camera instantly and gets a clean 2688x1520@20fps. So a fresh TCP connection works, but the stuck nvurisrcbin never recovers.

  1. Is a reconnect triggered on the software path? Pipeline?

Pipeline: nvurisrcbin (per camera) → nvstreammux (batch=2) → nvinfer. To be clear, the only thing I change is the source: I replace nvurisrcbin with rtspsrc protocols=tcp → rtph264depay → h264parse → avdec_h264 → nvvideoconvert → nvstreammux.

Yes, reconnects still happen on that path too — a blip still drops the socket. The difference is what happens next:

  • With plain rtspsrc, the error shows up on the bus (GST_MESSAGE_ERROR), so we can reset just that source (NULL → PLAYING) and the other camera keeps running.
  • With nvurisrcbin, nothing is posted to the bus. It just keeps looping try_send / pause / close on the dead socket forever. The pipeline still says PLAYING, so there is no way to know or react without restarting the whole thing.

That is the main problem: no error, and it never recovers.

  1. Camera IDR interval?

1 second — a keyframe every 20 frames at 20 fps (GOP = 20). Both cameras are the same model with the same 1 s IDR interval.

6

Can you share the detailed log?

7

Sharing the detailed log from a live run where the exit camera (src1) froze while the entry camera (src0) kept running at 20 fps on the same pipeline. Pipeline: nvurisrcbin ×2 → nvstreammux (batch=2) → nvinfer.

GStreamer debug log at the freeze (RTSP session layer):
WARN nvurisrcbin watch_source_status: No data from source since last 3 sec. Trying reconnection
WARN rtspsrc gst_rtspsrc_try_send: receive interrupted
WARN rtspsrc gst_rtspsrc_pause: PAUSE interrupted
WARN rtspsrc gst_rtspsrc_try_send: server closed connection
WARN rtspsrc gst_rtspsrc_try_send: error: Could not receive message. (Received end-of-file)
WARN rtspsrc gst_rtspsrc_close: error: Could not send message. (Received end-of-file)
WARN rtspsrc gst_rtspsrc_try_send: error: Could not send message. (Received end-of-file)
WARN rtspsrc gst_rtspsrc_pause: error: Could not send message. (Received end-of-file)
… this try_send / pause / close loop then repeats indefinitely …

Behaviour:

  • After the reconnect starts, rtspsrc sees server closed connection (EOF) and keeps looping try_send → pause → close on the dead socket forever.
  • Nothing is posted to the pipeline bus — no GST_MESSAGE_ERROR, no EOS. The pipeline stays PLAYING, so there’s no event to react to and no way to reset just that one source.
  • Our per-stage frame counters at the freeze: src1 (exit) stuck — the frames arriving from the camera dropped to 0 before decode, while src0 (entry) held 20 fps throughout.
  • The camera itself stays healthy: a fresh ffprobe -rtsp_transport tcp to the same camera connects instantly and pulls clean 2688×1520@20fps. Only the stuck nvurisrcbin never recovers.

This is the core problem: with plain rtspsrc the reconnect failure surfaces as GST_MESSAGE_ERROR on the bus, so we can reset that source (NULL→PLAYING) and keep the other camera running — but nvurisrcbin swallows it and loops silently.

Full GStreamer debug log attached.

gstreamer.log (50.4 KB)

nvurisrcbin_wedge.log (6.0 KB)

8

The EOF (server connection closed) from server means the RTSP session is closed by the RTSP server, rtspsrc will generate error message and send to bus.

The nvurisrcbin is open source. /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvurisrcbin. The bus error message is changed to warning message by gst_ds_nvurisrc_bin_handle_message function in gstdsnvurisrcbin.cpp. You can modify the code to keep the bus error messages.

If you don’t want to change the source code, you can set the “rtsp-reconnect-attempts” property of nvurisrcbin to a reasonable value and you will get the “rtsp-reconnect-attempts-exceed” message, then you can handle as you want. Please refer to /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app/deepstream_app.c (find the gst_nvmessage_is_reconnect_attempt_exceeded related code).

9

Update: tested both suggested fixes (propagate-rtsp-error + rtsp-reconnect-attempts) — neither reliably recovers the camera

Setup: Dual-camera (entry/exit) RTSP pipeline on Jetson AGX Thor, using nvurisrcbin to pull each camera over RTSP/TCP with hardware decode (NVDEC).

Background: As reported, after a brief RTSP network blip, nvurisrcbin’s reconnect logic can get permanently stuck — the camera freezes for good, with no error surfaced to the application (the real error is silently downgraded to a warning), so nothing can auto-recover it short of a full process restart.

As you suggested, two possible fixes:

  1. propagate-rtsp-error — a patch so the real error reaches the application instead of being downgraded to a warning, letting the app reset just that one camera’s source.

  2. rtsp-reconnect-attempts — cap the retry count so nvurisrcbin posts an attempt-exceeded message once exhausted, which the app can catch and act on.

What we tested: whether either fix, in isolation, actually lets the camera recover on its own after a network blip — without a full process restart. We ran each fix alone (disabling the other) against three blip durations (3s / 5s / 10s), forcing the blip with a temporary firewall rule blocking that camera’s RTSP port for the exact duration, then inspected the logs for the outcome.

Result: Neither fix reliably solves it. Both sometimes recovered on their own (via nvurisrcbin’s own internal retry — not because of either fix), and sometimes stayed wedged and required a full process restart, regardless of which fix was active or the blip duration. This inconsistency is what’s captured in the 6 attached log files.

Each log file contains: test metadata, environment/version info, sanitized config, exact repro steps, filtered GStreamer WARN/ERROR output, kernel log, and a result summary.

Let me know if i need to run with additional configurations or if there’s any specific variant/combo (blip duration, fix combination, decoder settings) that would help narrow this down further.

reconnect_test_5s_fix1_propagate-rtsp-error.log (88.6 KB)

reconnect_test_5s_fix2_rtsp-reconnect-attempts.log (25.0 KB)

reconnect_test_10s_fix1_propagate-rtsp-error.log (45.2 KB)

reconnect_test_10s_fix2_rtsp-reconnect-attempts.log (36.4 KB)

11

reconnect_test_3s_fix2_rtsp-reconnect-attempts.log (32.4 KB)

reconnect_test_3s_fix1_propagate-rtsp-error.log (96.9 KB)

12

How did you implement the camera recover?

13

Recovery is implemented at two levels — an in-pipeline per-source reset (what your two suggested fixes target), and an application-level watchdog + process restart as the actual fallback when that doesn’t clear the wedge.

Level 1 — per-source signal-driven reset (this is what Fix 1/Fix 2 hook into):
On the bus ELEMENT attempt-exceeded message, or on the bus ERROR surfaced via the propagate-rtsp-error patch, we call a reset function that cycles just that one nvurisrcbin element NULL → PLAYING (via GLib.idle_add, so it runs
off the main loop rather than inside the bus callback):
def _reset_hw_source(self, i, reason):
src = self._hw_sources.get(i)
src.set_state(Gst.State.NULL)
src.sync_state_with_parent()
This is the level our test logs show is unreliable — the element-level state cycle doesn’t consistently re-establish a fresh RTSP session after the internal reconnect logic has wedged.

Level 2 — frame-stall watchdog + full process restart (this is what actually recovers it in practice):
A watchdog checks wall-clock time since each source’s last delivered frame, independent of any GStreamer bus signal (it catches the case where frames silently stop but no ERROR/EOS is ever posted):
def _check_watchdog(self):
stalled = now - self._last_t.get(i, now)
if stalled > self.watchdog_timeout: # 45s
self._running = False
self.loop.quit()

Rather than trying to fix the wedged element in-place, this deliberately tears down the whole pipeline/process. Recovery is then handled outside our code entirely, by systemd:
Restart=always
RestartSec=5

This forces a completely fresh process — new pipeline, new elements, new TCP connections to both cameras — which is why it reliably clears the wedge when the element-level reset doesn’t.

In our 6-run test, the runs that “recovered” at ~47-51s in the logs are this watchdog+restart path firing, not either of the suggested fixes — the native nvurisrcbin reconnect only succeeded on its own in the runs that recovered well under 45s.

14

The source should be removed from the nvstreammux sink pad after the state reach NULL, and be added again to nvstreammux sink pad and be set to “PLAYING” state.

rtspsrc is just a RTSP client, it can’t reset the RTSP server if the RTSP server stops to work. The server side recovery should be done outside DeepStream pipeline.