Bug Report: nvmultiurisrcbin/nvurisrcbin/nvstreammux with HLS Streaming Issues

[Levi_Pereira]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
566.03
• Issue Type( questions, new requirements, bugs)
bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Environment

• Using GStreamer with NVIDIA DeepStream plugins
• Pipeline involves nvmultiurisrcbin/nvurisrcbin/nvstreammux with HLS streams

Issue Description

When using nvmultiurisrcbin/nvurisrcbin/nvstreammux with HLS streams, timing and playback issues occur, including pipeline freezing, depending on the configuration of live-source, max-batch-size and sync parameters.

Working Configuration without nvmultiurisrcbin/nvurisrcbin

The following pipeline works correctly:

gst-launch-1.0 souphttpsrc location=<redacted> ! hlsdemux ! decodebin ! videoconvert ! autovideosink

Note: the test URL(m3u8) can be shared privately with NVIDIA support team.

Problem Scenarios

Failed Pipeline Example

gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost \
batched-push-timeout=33333 max-batch-size=2 \
drop-pipeline-eos=1 live-source=1 \
uri-list=<redacted> width=720 height=480 \
! nvmultistreamtiler width=720 height=480 ! nveglglessink sync=0

Working but Restricted Pipeline

gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost \
batched-push-timeout=33333 max-batch-size=1 \
drop-pipeline-eos=1 live-source=0 \
uri-list=<redacted> width=720 height=480 \
! nvmultistreamtiler width=720 height=480 ! nveglglessink sync=1

Scenario 1: live-source=1 with max-batch-size >= num_source

• Configuration: nvmultiurisrcbin live-source=1 max-batch-size=3 nveglglessink sync=0 source=1

• Result: PTS is not respected

• Issue: Video plays ~1.5x faster and freezes at the next segment

Scenario 2: live-source=1 with max-batch-size = num_source

• Configuration: nvmultiurisrcbin live-source=1 max-batch-size=1 nveglglessink sync=0 source=1

• Result: PTS is not respected

• Issue: Video plays ~3.5x faster and freezes at the next segment

Scenario 3: live-source=0 with max-batch-size > num_source

• Configuration: nvmultiurisrcbin live-source=0 max-batch-size=5 nveglglessink sync=1 source=1

• Result: Similar timing issues

• Issue: Video plays ~1.5x faster and freezes

Working but Limited Scenario

• Configuration: live-source=0 with max-batch-size=num_source and nveglglessink sync=1

• Result: Works correctly

• Limitation: According to documentation, we need live-source=1 for dynamic pipelines (adding/removing sources on demand)

Documentation Reference

The documentation states that live-source=1 is required for dynamic pipelines:

“live-source need to be turned ON to allow batching of available buffers when number of sources is < max-batch-size configuration.”

Impact

• This bug prevents the proper use of nvmultiurisrcbin with HLS streams in dynamic pipeline scenarios where sources need to be added or removed on demand.

• Users are forced to use sync=1 in the sink because live-source=0 causes PTS (Presentation Time Stamps) to be disregarded, resulting in incorrect playback timing

• This creates a conflicting situation where:

• Using live-source=0 requires sync=1 to maintain proper timing

• But dynamic pipelines require live-source=1 according to documentation

• When using live-source=1, the PTS timing issues occur regardless of sync settings

Expected Behavior

The pipeline should maintain correct playback speed and timing when using live-source=1 max-batch-size > num_sources and sync=0 with HLS streams, allowing for dynamic source management while respecting PTS timing.

[Levi_Pereira]

Same as HLS framerate changed after nvstreammux

[junshengy]

1. For sink elements, sync=1 is necessary, otherwise the pipeline will run as quickly as possible, thus ignoring synchronization.
   2.I tested the following pipeline and it works fine. I set max-batch-size=10 and live-source=1 (run about 13 hours)

gst-launch-1.0 nvmultiurisrcbin port=9000 ip-address=localhost batched-push-timeout=33333 max-batch-size=10 drop-pipeline-eos=1 live-source=1 uri-list=xxxxxx.m3u8 width=720 height=480 ! nvmultistreamtiler width=720 height=480 ! nveglglessink

3.Set batch-size to 1 and let add/remove APIs update this logic cause pipeline slowdown and hence the maxBatchSize usage.This is a known issue，Use new streammux to avoid this issue

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 nvmultiurisrcbin port=9000 ip-address=localhost batched-push-timeout=33333 max-batch-size=1 drop-pipeline-eos=1 live-source=1 uri-list=http://10.19.226.168:8080/live/livestream.m3u8 width=720 height=480 ! nvmultistreamtiler width=720 height=480 ! nveglglessink

[Levi_Pereira]

Thank you for your time.
I wanted to share that the following pipeline command was successful in my testing:

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 nvmultiurisrcbin port=9000 ip-address=localhost batched-push-timeout=33333 max-batch-size=1 drop-pipeline-eos=1 live-source=1 uri-list=livestream.m3u8 width=720 height=480 ! nvmultistreamtiler width=720 height=480 ! nveglglessink

This configuration worked successfully while other variations I tried were problematic. I will be implementing this in my development tomorrow and will provide further updates. I’ve shared some additional test URLs privately for further testing.

[Levi_Pereira]

@junshengy I’m observing issues with HTTP live streaming:

“Redistributed latency…” warning and playback freezes occur on both VLC and GStreamer when using HTTP streams. This suggests a network latency issue rather than a player-specific problem. (Ignored)

Moreover, looks like when latency occurs in a single source, it affects all sources in the pipeline, causing cumulative delays across all sources over time. This cascading effect suggests possible resource contention or synchronization issues in the pipeline handling.

Are there any configurations available to prevent this cross-source latency propagation and maintain independent stream performance?

[Levi_Pereira]

Updates…
When using YouTube HLS streams, the issue doesn’t seem to occur.
However, when using my own streams, the problem manifests itself. I will investigate internally and return with more information.

Thank you for your time.

[junshengy]

Actually, I saw a similar situation, this may require reconnection.

You can refer to this FAQ, set the timeout, and then reconnect.

DeepStream SDK FAQ

37.[ALL_dGPU_APP] How to reconnect http stream in deepstream-app ? We take mjpeg over http as an example, add mjpeg-over-http=1 in [sourcex] group. Remove the uridecodebin, which will cause errors to be incorrectly passed in the pipeline. Monitor and reconnect the http stream in deepstream_source_bin.c Process EOS messages in bus_callback of deepstream_app.c Here is the entire patch: For apps-common diff --git a/sources/apps/apps-common/includes/deepstream_config_file_parser.h b/…

[Levi_Pereira]

I’m having trouble tracking down an issue while debugging. Looking at gst-nvurisrcbin , I noticed the RTSP implementation is well-built while HLS relies on uridecodebin. I made some improvements to better handle HLS streams, like using uridecodebin3 instead of uridecodebin. decodebin3 comes with enhanced HLS support, though I’m still evaluating its impact on my pipeline.

If the issue lies between nvurisrcbin and nvstreammux, visibility is limited since nvstreammux is closed source. It’s concerning that the entire pipeline freezes due to issues with a single stream.

I’ll check timeout feature.

Please let me know if you discover anything new about this issue.

[Levi_Pereira]

I’ve tested individual streams without issues - although they sometimes freeze for a few seconds, the pipeline self-corrects. However, when I combine streams with different latency characteristics in the same pipeline (some with latency, others without), the pipeline starts experiencing issues. In a pipeline with 12 streams where 6 freeze randomly, from the user’s perspective, the pipeline appears constantly frozen because each freeze lasts about 3 seconds.

I believe the main issue here is that latency from one stream affects the entire pipeline. It seems the freezing occurs in the streammux, which waits for the frozen stream to recover. While I can’t confirm this definitively, this is my observation of the behavior.

The core problem appears to be that individual stream latency impacts the whole pipeline, with the streammux waiting for frozen streams to recover before proceeding. This is my current understanding of the situation.

I’m having difficulty understanding exactly which component causes the entire pipeline to freeze when latency occurs. Is it the streammux that’s blocking the pipeline, or is there another component responsible for this behavior? Any insights would be appreciated.

[junshengy]

@Levi_Pereira

If the issue lies between nvurisrcbin and nvstreammux, visibility is limited since nvstreammux is closed sour

The new nvstreammux is open source, you can refer to this

/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmultistream2

@Levi_Pereira

The core problem appears to be that individual stream latency impacts the whole pipeline, with the streammux waiting for frozen streams to recover before proceeding. This is my current understanding of the situation.

nvstreammux should not cause freezing. I tested nvmultiurisrcbin and added a file url and a rtsp camera. Then I disconnected the rtsp camera from the network and the pipeline did not get stuck.

[Levi_Pereira]

@junshengy This may be a direction
Adaptive Demuxers for DASH, HLS and Smooth Streaming

[Levi_Pereira]

Actually, I saw a similar situation, this may require reconnection.
You can refer to this FAQ, set the timeout, and then reconnect.

I’m using Python to manage the pipeline. Can we do something similar code example with nvurisrcbin/nvmultiurisrcbin?

I’m experiencing a challenging problem with HLS streaming in my pipeline:

1. Stream Instability Symptoms:

• When an HLS stream stops functioning, the entire pipeline becomes unresponsive
• Upon stream recovery, the pipeline seems to execute as if sync=false is active
• This behavior is particularly disruptive and unexpected

2. Current Configuration:

• I can start and manage 8 streams successfully under stable conditions
• Stream addition and removal work normally during stable periods
• However, if any single stream becomes unstable, it impacts the entire pipeline’s performance

3. Attempted Solutions:

• I’ve been working on modifying nvurisrcbin to handle HLS streams more robustly
• Despite efforts, I haven’t yet achieved a satisfactory solution
• The streams perform well during the day, but occasional latency disrupts the entire pipeline

I’m seeking guidance or potential implementations of an nvurisrcbin version that can more gracefully manage HLS stream instabilities. Any insights or code examples would be immensely appreciated.

[junshengy]

You may not be able to use the FAQ code snippet directly, but I think reconnecting when stuck is still a solution

nvurisrcbin only supports rtspsrc reconnection, but after some modifications hls reconnection can also be achieved

It may not be possible to complete this work using only python, and the code of nvurisrcbin element needs to be modified.

The following functions need to be modified in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp

1.rtspsrc_monitor_probe_func
2.watch_source_status
3.reset_source_pipeline

[Levi_Pereira]

Thank you for the guidance on the reconnection implementation.

I have modified the nvurisrcbin element following the suggested approach to handle both RTSP and HLS reconnection scenarios.

While testing the implementation, I’ve noticed an additional challenge: when any stream in the pipeline experiences latency or freezes, it causes the entire pipeline to freeze. This blocking behavior affects the overall pipeline performance.

Could you please:

1. Review if my reconnection implementation approach is correct
2. Provide guidance on handling the pipeline freezing issue, particularly when individual streams experience latency

If there are any best practices or recommended patterns for handling stream latency without affecting the entire pipeline, I would greatly appreciate your insights.

gstdsnvurisrcbin.cpp.zip (1.5 KB)

[Levi_Pereira]

HLS Flow

[junshengy]

@Levi_Pereira

If there are any best practices or recommended patterns for handling stream latency without affecting the entire pipeline, I would greatly appreciate your insights.

gstdsnvurisrcbin.cpp.zip (1.5 KB)

This code should not work. I have a simple implementation below, but it took me a while and I still encountered some problems. hlsdemux encountered an error.

You can improve this reference implementation

diff --git a/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp b/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp
index dd78cf2..92b4677 100644
--- a/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp
+++ b/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.cpp
@@ -73,6 +73,8 @@
 #define GST_TYPE_V4L2_VID_CUDADEC_MEM_TYPE (gst_video_cudadec_mem_type ())
 #define DEFAULT_CUDADEC_MEM_TYPE 0
 
+static gboolean watch_source_status (gpointer data);
+
 template <class T>
 class GstObjectUPtr:public std::unique_ptr<T, void (*)(T *)>
 {
@@ -370,6 +372,24 @@ gst_ds_nvurisrc_bin_handle_message (GstBin * bin, GstMessage * message)
     GST_MESSAGE_TYPE (message) = GST_MESSAGE_WARNING;
   }
 
+  if (ubin->is_hls 
+      && ubin->source_watch_id 
+      && GST_MESSAGE_TYPE (message) == GST_MESSAGE_ERROR) {
+    gchar *debug = NULL;
+    GError *error = NULL;
+    gst_message_parse_error (message, &error, &debug);
+    if (g_strrstr (debug, "reason not-linked")) {
+      printf("ignore reason not-linked \n");
+      gst_message_unref (message);
+      return;
+    }
+
+    GstElement *msg_src_elem = GST_ELEMENT(GST_MESSAGE_SRC(message));
+    const gchar *element_name = GST_ELEMENT_NAME(msg_src_elem);
+    printf("ignore hls error %s \n", element_name);
+    GST_MESSAGE_TYPE (message) = GST_MESSAGE_WARNING;
+  }
+
   /* Allow decoded pads to be unlinked. */
   if ((GST_IS_OBJECT (ubin->cap_filter) && GST_IS_OBJECT (ubin->aqueue))
       && (GST_MESSAGE_SRC (message) == GST_OBJECT (ubin->cap_filter)
@@ -1141,6 +1161,12 @@ decodebin_child_added (GstChildProxy * child_proxy, GObject * object,
   if ((g_strrstr (name, "h264parse") == name) ||
       (g_strrstr (name, "h265parse") == name)) {
     g_object_set (object, "config-interval", -1, NULL);
+    if (bin->config->rtsp_reconnect_interval_sec > 0) {
+      printf("add monitor \n");
+      NVGSTDS_ELEM_ADD_PROBE (bin,
+          GST_ELEMENT(object), "src", rtspsrc_monitor_probe_func,
+          GST_PAD_PROBE_TYPE_BUFFER | GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, bin);
+    }
   }
   if (g_strrstr (name, "fakesink") == name) {
     g_object_set (object, "enable-last-sample", FALSE, NULL);
@@ -1404,6 +1430,24 @@ populate_uri_bin_audio (GstDsNvUriSrcBin * nvurisrcbin, GstPad * pad)
   return TRUE;
 }
 
+static void
+deep_element_added_callback (GstBin * self, GstBin * sub_bin,
+    GstElement * element, gpointer user_data)
+{
+  GstDsNvUriSrcBin *bin = (GstDsNvUriSrcBin *) user_data;
+  GstElementFactory *factory = gst_element_get_factory (element);
+  const gchar *factory_name = gst_plugin_feature_get_name (factory);
+
+  if (!g_strcmp0 (factory_name, "souphttpsrc")) {
+    if (bin->config->rtsp_reconnect_interval_sec > 0) {
+      printf("add monitor \n");
+      NVGSTDS_ELEM_ADD_PROBE (bin,
+          GST_ELEMENT(element), "src", rtspsrc_monitor_probe_func,
+          GST_PAD_PROBE_TYPE_ALL_BOTH, bin);
+    }
+  }
+}
+
 static gboolean
 populate_uri_bin (GstDsNvUriSrcBin * nvurisrcbin)
 {
@@ -1417,6 +1461,21 @@ populate_uri_bin (GstDsNvUriSrcBin * nvurisrcbin)
     return FALSE;
   }
 
+  nvurisrcbin->is_hls = config->uri && 
+      (g_str_has_prefix(config->uri, "http://") ||
+       g_str_has_prefix(config->uri, "https://")) &&
+       g_strrstr(config->uri, ".m3u8");
+
+  if (nvurisrcbin->is_hls && config->rtsp_reconnect_interval_sec > 0) {
+    printf("add watch_source_status %s \n", config->uri);
+    nvurisrcbin->source_watch_id =
+        g_timeout_add (1000, watch_source_status, nvurisrcbin);
+    g_mutex_lock (&nvurisrcbin->bin_lock);
+    gettimeofday (&nvurisrcbin->last_buffer_time, NULL);
+    gettimeofday (&nvurisrcbin->last_reconnect_time, NULL);
+    g_mutex_unlock (&nvurisrcbin->bin_lock);
+  }
+
   if (config->uri && g_str_has_prefix (config->uri, "rtsp://")) {
     configure_source_for_ntp_sync (nvurisrcbin->src_elem);
   }
@@ -1430,6 +1489,8 @@ populate_uri_bin (GstDsNvUriSrcBin * nvurisrcbin)
       G_CALLBACK (decodebin_child_added), nvurisrcbin);
   g_signal_connect (G_OBJECT (nvurisrcbin->src_elem), "source-setup",
       G_CALLBACK (cb_sourcesetup), nvurisrcbin);
+  // g_signal_connect (G_OBJECT (nvurisrcbin->src_elem), "deep-element-added",
+  //       G_CALLBACK (deep_element_added_callback), nvurisrcbin);
 
   gst_bin_add (GST_BIN (nvurisrcbin), nvurisrcbin->src_elem);
 
diff --git a/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.h b/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.h
index 11fe5be..d16a155 100644
--- a/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.h
+++ b/sources/gst-plugins/gst-nvurisrcbin/gstdsnvurisrcbin.h
@@ -101,6 +101,7 @@ typedef struct _GstDsNvUriSrcBin
 
   gboolean video_elem_populated;
   gboolean audio_elem_populated;
+  gboolean is_hls;
 } GstDsNvUriSrcBin;
 
 typedef struct _GstDsNvUriSrcBinClass

[junshengy]

I’m not sure why the entire pipeline is stuck. I tested the unavailable source using nvmultiurisrcbin and it didn’t get stuck. Did you use nvstreamdemux?

[Levi_Pereira]

News:
I have successfully implemented the HLS source in deepstream, similar to the existing RTSP source implementation. However, the HLS source has different characteristics compared to the RTSP source, and using the uridecodebin is not the best choice for maintaining pipeline stability. To ensure stability, we need to configure and have control over the following two components: souphttpsrc and hlsdemux.

I had to implement the following functions:

• populate_hls_bin
• cb_newpad_hlsdemux
• populate_hls_bin_video (audio implementation is pending as I’m not currently using audio)
  The rtspsrc_monitor_probe_func was placed at hlsdemux.

Removing these components causes a segmentation fault because, unlike RTSP, the souphttpsrc and hlsdemux components can get stuck waiting for download data. If the data is not removed in the correct order, the system aborts with a segmentation fault. This is applicable to both reset_source_pipeline and s_nvmultiurisrcbincreator_remove_source_impl functions, which need to handle these situations.

I encountered segmentation fault issues even when using the original implementation with the uridecodebin. The stop signal propagates through the pipeline, but if there’s no certainty that the element has actually stopped, it aborts the entire process.

However, I have resolved the implementation problem for HLS that didn’t exist before. Using stable HLS streams like YouTube, the uridecodebin works well. But if poor HLS streams are used, the system is not prepared, and users may encounter many problems.

It would be beneficial to update to GStreamer 1.24 or higher to leverage the hlsdemux2 element, which offers adaptive streaming improvements and Low-Latency HLS (LL-HLS) support.

Additionally, implementing the smart_recorder has become much easier now, as we only need to create the tee elements and insert them as done for the RTSP source.

The pipeline has become more stable, but I still need to investigate why the pipeline gets stuck if a stream freezes. I will be working on this next.

I removed tsdemux since hlsdemux can handle with mpegts.
s_nvmultiurisrcbincreator_remove_source_impl flow:

[Levi_Pereira]

Note

I encountered segmentation fault issues even when using the original implementation with the uridecodebin. The stop signal propagates through the pipeline, but if there’s no certainty that the element has actually stopped, it aborts the entire process.

This is related to DeepStream Coredump on Source Removal with Gst-nvmultiurisrcbin

[Levi_Pereira]

I’m not sure why the entire pipeline is stuck. I tested the unavailable source using nvmultiurisrcbin and it didn’t get stuck. Did you use nvstreamdemux ?

I discovered that the pipeline stalling issue was related to sync configurations rather than the nvstreamdemux. The problem was resolved by:

1. Optimizing sync and async settings across sinks:

   • Set max-lateness=-1
   • Disabled QoS where not critical
   • Adjusted throttle-time to 0
   • Set async=false to ensure synchronous state changes
   • Maintained sync=true for proper frame timing

2. Added queues before sinks to handle jitter:

   • Configured leaky=2 for downstream dropping
   • Set appropriate buffer sizes
   • Enabled flush-on-eos for cleanup

After these adjustments, the pipeline handles unavailable sources smoothly without stalling. The queues help absorb timing variations and prevent backpressure from affecting the entire pipeline.