[DeepStream 7.1 / Jetson Orin Nano / JetPack 6.x] Pipeline freeze with old nvstreammux and multiple live sources
Scenario
I am running a DeepStream pipeline on Jetson Orin Nano devices with JetPack 6.x.
The issue has been reproduced on both hardware configurations:
| Hardware | JetPack / L4T |
|---|---|
| Jetson Orin Nano 4GB | JetPack 6.x / L4T 36.4.3 |
| Jetson Orin Nano 8GB | JetPack 6.x / L4T 36.4.3 |
Software stack:
| Component | Version |
|---|---|
| DeepStream | 7.1 |
| GStreamer | 1.20.3 |
| OS | Ubuntu 22.04 |
| Stream muxer | Old nvstreammux |
| Tracker | nvtracker with NvDCF tracker |
The same application logic and equivalent pipeline configuration were stable on:
DeepStream 6.3 / JetPack 5.x / GStreamer 1.16.3
The freeze started appearing after moving to:
DeepStream 7.1 / JetPack 6.x / GStreamer 1.20.3
Use case
The application processes multiple live video sources at the same time.
The sources are mixed protocols:
- HTTP streams using
souphttpsrc - RTSP streams using
rtspsrc
Each source is decoded, converted, batched with old nvstreammux, processed by inference and tracking, then demuxed again and sent to an appsink.
Simplified pipeline:
[souphttpsrc / rtspsrc]
|
[decodebin]
|
[nvvideoconvert]
compute-hw=1
|
[old nvstreammux]
live-source=TRUE
batch-size=N
|
[nvinfer]
|
[nvtracker - NvDCF]
|
[nvstreamdemux]
|
[queue] -> [nvvideoconvert] -> [appsink]
There is no queue between nvstreammux and nvinfer.
Problem description
After a variable runtime, usually within a few hours, the pipeline enters a permanent frozen state.
The issue has been observed when more than two live sources are active simultaneously.
Observed behavior:
- all streams freeze at the same time;
- all queues downstream of
nvstreamdemuxfill up to their maximum size almost simultaneously; - after the queues are full, no more frames are produced;
appsinkcallbacks are no longer called;- no useful errors or messages are posted on the GStreamer bus;
- the issue does not seem related to a single source, because it happens simultaneously across HTTP and RTSP streams.
Example of the observed behavior:
[T+0s] queue_0: 0 buffers / 0ms
[T+3s] queue_0: 75 buffers / 5000ms
[T+3s] queue_1: 75 buffers / 5000ms
[T+3s] queue_2: 75 buffers / 5000ms
# From this point on, there is no further pipeline activity
Since all queues saturate at the same time, across different source protocols, the issue appears to be in the shared part of the pipeline:
nvstreammux -> nvinfer -> nvtracker -> nvstreamdemux
rather than in a single source branch.
Diagnostics performed
I added a GST_PAD_PROBE_TYPE_BUFFER probe on the src pad of nvstreammux.
During normal operation:
the probe receives batches every 40–80 ms
Occasionally, small gaps are observed:
200–300 ms
but the pipeline recovers correctly.
When the freeze occurs:
the probe on the nvstreammux src pad completely stops receiving buffers
The pipeline formally remains in PLAYING state, but nvstreammux no longer produces output batches.
Also, calling the following from a separate thread:
gst_element_set_state(pipeline, GST_STATE_NULL);
stops the pipeline correctly in around 250 ms.
This suggests that it is not a full GStreamer state deadlock, but rather a stall in the streaming data path or in batch production.
Workarounds already applied
The following workarounds have already been applied.
1. nvvideoconvert with compute-hw=1
Without this setting, the pipeline fails with errors like:
nvbufsurftransform_copy.cpp: Failed in mem copy
Applied workaround:
g_object_set(nvvideoconvert, "compute-hw", 1, NULL);
2. Disabled souphttpsrc timeout
To avoid disconnections caused by the default libsoup3 timeout:
g_object_set(souphttpsrc, "timeout", 0, NULL);
3. Live queues configured as leaky
To avoid blocking live source threads during downstream stalls:
g_object_set(vqueue, "leaky", 2, NULL);
4. Avoided gst_element_release_request_pad() on old nvstreammux while the pipeline is PLAYING
I observed that calling gst_element_release_request_pad() on old nvstreammux while the pipeline is running can cause a deadlock.
Applied workaround:
- I do not release request pads while the pipeline is running;
- when reconnecting a source, I reuse the existing pad if it already exists.
Questions
-
Are there any known regressions in DeepStream 7.1 / JetPack 6.x with old
nvstreammux, multiple live sources, andnvtrackerusing NvDCF? -
Are there known cases where old
nvstreammuxstops producing output batches while the pipeline remains inPLAYINGstate? -
With DeepStream 7.1 on Jetson, what are the recommended settings for old
nvstreammuxin live multi-source pipelines? -
Can
nvtrackerwith NvDCF introduce backpressure or stalls that propagate back tonvstreammuxin this kind of pipeline? -
What additional logs, probes, or debug environment variables do you recommend to diagnose this kind of freeze?
Additional notes
- The same scenario was stable on DeepStream 6.3 / JetPack 5.x.
- The issue has been reproduced on both Jetson Orin Nano 4GB and Jetson Orin Nano 8GB, both with L4T 36.4.3.
- The issue occurs with both HTTP and RTSP sources.
- The freeze is global and simultaneous across all streams.
- Thermal throttling has been excluded using
tegrastats. - Memory exhaustion has been excluded: RAM usage is stable at the time of the freeze.