Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
570.181
• Issue Type( questions, new requirements, bugs)
bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Summary
Component: nvstreammux (gst-nvmultistream2)
Version: DeepStream 7.1
Issue: HLS source plays too fast on first addition, normal after remove/re-add
Severity: High - Affects live HLS streaming applications
Reproducibility: 100% reproducible
Problem Description
When adding an HLS source to nvstreammux for the first time, video playback is significantly faster than expected. After removing and re-adding the same source, playback becomes normal.
Observed Behavior
- First HLS source addition: Video plays 5-10x faster than real-time
- Remove and re-add same source: Video plays at correct speed
- Affects all sink types: nveglglessink, fakesink, udpsink
- Independent of sync property: Occurs with both sync=true and sync=false
Pipeline Configuration
USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 \
nvmultiurisrcbin \
port=9000 \
ip-address=localhost \
max-batch-size=16 \
drop-pipeline-eos=1 \
live-source=1 \
! nvmultistreamtiler \
width=1280 \
height=720 \
rows=1 \
columns=1 \
square-seq-grid=1 \
! nvdsosd \
process-mode=0 \
! nveglglessink \
sync=false \
async=false \
qos=false
Steps to Reproduce
- Start pipeline with HLS source
- Observe fast playback (video appears to fast-forward)
- Remove source from nvmultiurisrcbin
- Re-add same HLS source
- Observe normal playback speed
Debug Evidence
Added Debug Logging
Added instrumentation to gstnvstreammux.cpp and gstnvstreammux_pads.cpp to observe timing behavior:
Code to Identify First vs Subsequent Source Addition
// In gstnvstreammux.cpp - Detect first source addition
if(mux->cur_frame_pts==0)
{
mux->pts_offset = 0; // First source uses 0 offset
mux->cur_frame_pts=1;
mux->helper->set_pts_offset(0);
g_print("[EXPERIMENT] FIRST SOURCE - pts_offset=%lu\n", mux->pts_offset);
}
else
{
mux->pts_offset = mux->synch_buffer->GetCurrentRunningTime(); // Subsequent sources use running time
mux->helper->set_pts_offset(0);
g_print("[EXPERIMENT] SUBSEQUENT SOURCE - pts_offset=%lu\n", mux->pts_offset);
}
Code to Measure Time Between Buffer Pushes
// In gstnvstreammux_pads.cpp - Measure push timing
static GstClockTime last_push_time = 0;
GstClockTime current_time = gst_clock_get_time(gst_element_get_clock(GST_ELEMENT(mux)));
GstClockTime time_since_last_push = 0;
if (last_push_time > 0) {
time_since_last_push = current_time - last_push_time;
}
g_print("[EXPERIMENT] PTS push - original=%lu, offset=%lu, final=%lu, time_since_last=%lu ms\n",
original_pts, mux->pts_offset, pts, time_since_last_push / 1000000);
last_push_time = current_time;
First Addition Logs (PROBLEMATIC)
[HLS_DEBUG] Source state change - stream_index=0, state: 0->0
[HLS_DEBUG] Buffer received - stream_index=0, original_pts=0, duration=66666666
[EXPERIMENT] Source IDLE - stream_index=0, cur_frame_pts=0
[EXPERIMENT] Pipeline state=PLAYING, pending=VOID_PENDING
[EXPERIMENT] base_time=34349390902760, current_running_time=6521310289
[EXPERIMENT] sync_inputs=0, peer_latency_live=0, no_pipeline_eos=1
[EXPERIMENT] FIRST SOURCE - pts_offset=6521310289
[HLS_DEBUG] Source state change - stream_index=0, state: 0->2
[HLS_DEBUG] Buffer received - stream_index=0, original_pts=67000000, duration=66666666
[HLS_DEBUG] Buffer received - stream_index=0, original_pts=133000000, duration=66666666
[EXPERIMENT] PTS push - original=66666666, offset=6521310289, final=6587976955, time_since_last=0 ms
[HLS_DEBUG] Buffer received - stream_index=0, original_pts=200000000, duration=66666666
[EXPERIMENT] PTS push - original=133333332, offset=6521310289, final=6654643621, time_since_last=0 ms
[EXPERIMENT] PTS push - original=199999998, offset=6521310289, final=6721310287, time_since_last=0 ms
[EXPERIMENT] PTS push - original=266666664, offset=6521310289, final=6787976953, time_since_last=4 ms
[EXPERIMENT] PTS push - original=333333330, offset=6521310289, final=6854643619, time_since_last=6 ms
Analysis: 5 frames pushed with intervals: 0ms, 0ms, 0ms, 4ms, 6ms
Expected: ~100ms intervals for 10fps source
Second Addition Logs (WORKING)
[HLS_DEBUG] Source state change - stream_index=1, state: 2->0
[HLS_DEBUG] Buffer received - stream_index=1, original_pts=0, duration=66666666
[EXPERIMENT] Source IDLE - stream_index=1, cur_frame_pts=1
[EXPERIMENT] Pipeline state=PLAYING, pending=VOID_PENDING
[EXPERIMENT] base_time=34349390902760, current_running_time=28288360200
[EXPERIMENT] sync_inputs=0, peer_latency_live=0, no_pipeline_eos=1
[EXPERIMENT] SUBSEQUENT SOURCE - pts_offset=28288360200
[HLS_DEBUG] Source state change - stream_index=1, state: 0->2
[HLS_DEBUG] Buffer received - stream_index=1, original_pts=67000000, duration=66666666
[HLS_DEBUG] Buffer received - stream_index=1, original_pts=134000000, duration=66666666
[EXPERIMENT] PTS push - original=66666666, offset=28288360200, final=28355026866, time_since_last=100 ms
[HLS_DEBUG] Buffer received - stream_index=1, original_pts=201000000, duration=66666666
[EXPERIMENT] PTS push - original=133333332, offset=28288360200, final=28421693532, time_since_last=100 ms
[EXPERIMENT] PTS push - original=199999998, offset=28288360200, final=28488360198, time_since_last=100 ms
Analysis: 3 frames pushed with correct intervals: 100ms, 100ms, 100ms
Expected: ~100ms intervals for 10fps source ✓
Key Observations
Pipeline State Comparison
Both additions show identical pipeline state:
- Pipeline state: PLAYING
- base_time: 34349390902760 (identical)
- sync_inputs: 0 (identical)
- peer_latency_live: 0 (identical)
- no_pipeline_eos: 1 (identical)
Timing Behavior Difference
First addition: time_since_last=0ms, 0ms, 4ms, 6ms (burst)
Second addition: time_since_last=100ms, 100ms, 100ms (smooth)
Original Code Behavior
The original nvstreammux code (without modifications) behaves as follows:
// In gstnvstreammux.cpp - Original behavior
if((mux->helper->get_pad_state(sinkPad) == SOURCE_STATE_IDLE) && (!mux->isAudio) )
{
if(mux->cur_frame_pts==0)
{
mux->pts_offset = 0; // First source uses 0 offset
mux->cur_frame_pts=1;
mux->helper->set_pts_offset(0);
}
else
{
mux->pts_offset = mux->synch_buffer->GetCurrentRunningTime(); // Subsequent sources use running time
mux->helper->set_pts_offset(0);
}
}
Note: This code difference exists, but the timing issue persists regardless of offset value.
Impact
User Experience
- Video appears to “fast forward” for 1-5 seconds
- Then pauses/buffers while waiting for next segment
- Creates unprofessional viewing experience
Tested Workarounds
1. Adjusting pts_offset
- ✗ Setting first source offset to 0 (original behavior)
- ✗ Setting first source offset to current_running_time
- ✗ Setting first source offset to current_running_time + 20s
- Result: No change in timing behavior (
time_since_laststill shows burst)
2. Modifying batched-push-timeout
- ✗ Various timeout values (33ms, 100ms, 200ms)
- Result: May delay burst but doesn’t eliminate it
3. Different sink configurations
- ✗ nveglglessink sync=true
- ✗ nveglglessink sync=false
- ✗ fakesink
- ✗ udpsink
- Result: All sinks affected, confirming issue is upstream
Request
Please investigate why nvstreammux exhibits different timing behavior between:
- First HLS source addition (burst timing)
- Subsequent HLS source additions (smooth timing)
The issue appears to be related to buffer processing timing rather than PTS calculation, as evidenced by the time_since_last measurements.