DeepStream parallel pipeline frame synchronization is incorrect when using the new streammux and a non-zero PGIE interval

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) 4090
• DeepStream Version 7.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 535.274.02
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Context:
I’m running the DeepStream parallel pipeline based on the following repository:
🔗 https://github.com/NVIDIA-AI-IOT/deepstream_parallel_inference_app

Setup:

  • Input: multiple .mp4 videos

  • Model type: Triton Inference Server (Yolov8)

  • Pipeline: 2 branches, each with one PGIE model (same model, but different IDs in the config file, e.g., id: 103 and id: 104)

Issue Description:
When interval = 0, all outputs are correct.
However, when I set interval > 0 (for example, interval = 5), I noticed frame synchronization issues in the metamux output.

In the metamux src_pad probe callback, only frames where frame_id % 6 == 0 should contain object detection bounding boxes. But in reality, I observe that some sources produce bounding boxes on frames where frame_id equals (multiple of 6) - 1.

This means that either metamux or another element in the pipeline outputs detection results one frame early, resulting in misaligned object detection timing between branches.

Below is an example of the erroneous JSON output (I export the outputs of all frame_idx values to make debugging easier):

{"video_url": 0, "frame_id": 0, "detections": [{"xyxy": [290, 441, 476, 866], "confidence": 0.93603515625, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [0, 390, 200, 1029], "confidence": 0.90234375, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [123, 465, 207, 768], "confidence": 0.7919921875, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [463, 486, 492, 558], "confidence": 0.62158203125, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [433, 477, 459, 548], "confidence": 0.57275390625, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [611, 495, 715, 569], "confidence": 0.8642578125, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [544, 472, 599, 518], "confidence": 0.7919921875, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [588, 478, 661, 531], "confidence": 0.6513671875, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}]}
{"video_url": 1, "frame_id": 0, "detections": [{"xyxy": [120, 82, 378, 592], "confidence": 0.92236328125, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [459, 116, 534, 531], "confidence": 0.5361328125, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [120, 82, 378, 592], "confidence": 0.92236328125, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [459, 116, 534, 531], "confidence": 0.5361328125, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}]}
{"video_url": 2, "frame_id": 0, "detections": [{"xyxy": [1265, 897, 1482, 1078], "confidence": 0.896484375, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [869, 630, 1076, 903], "confidence": 0.85205078125, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [871, 404, 977, 560], "confidence": 0.8310546875, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [450, 327, 563, 432], "confidence": 0.8017578125, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [625, 137, 676, 193], "confidence": 0.73876953125, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [432, 136, 494, 196], "confidence": 0.66552734375, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [655, 0, 685, 26], "confidence": 0.662109375, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [609, 48, 643, 83], "confidence": 0.53125, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}]}
{"video_url": 3, "frame_id": 0, "detections": [{"xyxy": [707, 297, 729, 350], "confidence": 0.76513671875, "class_id": 0, "class_name": "person", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [384, 315, 521, 426], "confidence": 0.91845703125, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [192, 311, 343, 408], "confidence": 0.9091796875, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [362, 333, 396, 361], "confidence": 0.79541015625, "class_id": 2, "class_name": "car", "frame_id": 0, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}]}
{"video_url": 0, "frame_id": 1, "detections": []}
{"video_url": 2, "frame_id": 1, "detections": []}
{"video_url": 3, "frame_id": 1, "detections": []}
{"video_url": 0, "frame_id": 2, "detections": []}
{"video_url": 1, "frame_id": 1, "detections": []}
{"video_url": 2, "frame_id": 2, "detections": []}
{"video_url": 3, "frame_id": 2, "detections": []}
{"video_url": 0, "frame_id": 3, "detections": []}
{"video_url": 1, "frame_id": 2, "detections": []}
{"video_url": 2, "frame_id": 3, "detections": []}
{"video_url": 3, "frame_id": 3, "detections": []}
{"video_url": 0, "frame_id": 4, "detections": []}
{"video_url": 1, "frame_id": 3, "detections": []}
{"video_url": 2, "frame_id": 4, "detections": []}
{"video_url": 3, "frame_id": 4, "detections": []}
{"video_url": 0, "frame_id": 5, "detections": []}
{"video_url": 1, "frame_id": 4, "detections": []}
{"video_url": 2, "frame_id": 5, "detections": []}
{"video_url": 3, "frame_id": 5, "detections": []}
{"video_url": 0, "frame_id": 6, "detections": [{"xyxy": [302, 445, 479, 857], "confidence": 0.91650390625, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [1, 391, 195, 1004], "confidence": 0.83544921875, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [118, 464, 210, 771], "confidence": 0.67578125, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [463, 484, 491, 554], "confidence": 0.625, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [435, 479, 461, 552], "confidence": 0.60595703125, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [612, 496, 719, 574], "confidence": 0.86474609375, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [544, 473, 602, 516], "confidence": 0.75927734375, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [589, 478, 662, 531], "confidence": 0.65869140625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}]}
{"video_url": 1, "frame_id": 5, "detections": [{"xyxy": [96, 96, 293, 652], "confidence": 0.91943359375, "class_id": 0, "class_name": "person", "frame_id": 5, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}, {"xyxy": [452, 61, 641, 538], "confidence": 0.8759765625, "class_id": 0, "class_name": "person", "frame_id": 5, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 103}]}
{"video_url": 2, "frame_id": 6, "detections": [{"xyxy": [867, 635, 1084, 953], "confidence": 0.8701171875, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [404, 381, 536, 508], "confidence": 0.85400390625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [867, 411, 977, 570], "confidence": 0.84619140625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [625, 154, 678, 212], "confidence": 0.7509765625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [610, 57, 645, 92], "confidence": 0.67431640625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [433, 135, 494, 195], "confidence": 0.64794921875, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [656, 1, 688, 30], "confidence": 0.63134765625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}]}
{"video_url": 3, "frame_id": 6, "detections": [{"xyxy": [708, 296, 731, 350], "confidence": 0.802734375, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [384, 315, 519, 425], "confidence": 0.91845703125, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [195, 311, 344, 408], "confidence": 0.9072265625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [363, 333, 397, 361], "confidence": 0.80712890625, "class_id": 2, "class_name": "car", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}]}
{"video_url": 0, "frame_id": 7, "detections": []}
{"video_url": 1, "frame_id": 6, "detections": [{"xyxy": [74, 102, 294, 645], "confidence": 0.91259765625, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}, {"xyxy": [453, 64, 678, 534], "confidence": 0.888671875, "class_id": 0, "class_name": "person", "frame_id": 6, "using_phone_scores": 0.0, "attach_phone_scores": 0.0, "unique_component_id": 104}]}
{"video_url": 2, "frame_id": 7, "detections": []}

error: “video_url”: 1, “frame_id”: 5 => Pay attention to the unique_component_id — at this point, frame_5 of video 1 contains the output from one branch.

This issue occurs randomly on certain frames, but once an error happens at a specific frame_idx, the bounding boxes for that video_url start appearing every +6 frames from that point onward. When running the pipeline multiple times, the issue happens intermittently — sometimes it’s correct, but most of the time the results are misaligned.

I ran the sample using the DeepStream C version and encountered this issue with the new streammux. With the old streammux, the problem didn’t seem to occur, but the pipeline would hang or behave incorrectly when one of the sources reached EOS, since the new streammux includes a mechanism to handle EOS signals from individual sources.

After that, I reimplemented the pipeline using the Python version via pyds. I’ll include the pipeline diagram and configuration files below, but the same issue still occurs — even when running only 4 sources.

Next, I tested on a machine with an RTX 5090 GPU and DeepStream 8.0. The issue did not appear when running a small number of videos. However, when scaling up (for example, branch 1 running 40 videos and branch 2 running 2 videos), the problem reappeared.

Do you have any idea what might be causing this issue? Thank you.

parallel_issues.zip (80.7 MB)

3

You’ve described too many issues; let’s narrow it down.

1.deepstream_parallel_inference_app + new nvstreammux
2.deepstream_parallel_inference_app + legacy nvstreammux
3.User-customized pyds_parallel_inference_app.

We usually only handle issues related to the official demo and the latest version.

Could you provide a configuration file using the sample code below, along with new streammux and DeepStream-8.0, and record a video of the error so we can understand the problem?

Additionally, I want to clarify that the interval parameter controls the batch inference interval, not the frame .

Because nvstreammux doesn’t necessarily collect all sources into a batch, as this depends on factors like frame rate and network conditions.

4

I tested multiple versions to see whether this issue occurs, and I want to ask specifically about newstreammux and pgie interval > 0.
You mentioned: “Because nvstreammux doesn’t necessarily collect all sources into a batch, as this depends on factors like frame rate and network conditions.”

How can I ensure that I only process frames where frame_idx % 6 == 0 for each video if the streammux doesn’t pull frames from each video source evenly every cycle?

When running a single branch (not in parallel), it seems that this issue does not occur.

Can you try to reproduce the issue to see whether it occurs on your side? If newstreammux is causing the problem, is it possible to run with the old streammux instead?
When I tested with the old streammux, the pipeline became stuck, and when one of the sources reached EOS, the frame_idx values became incorrect.
I tested with video mp4

I tested this on DeepStream 7.0. If you can reproduce the issue on DeepStream 8.0 (In version 8.0, the issue happens less frequently, and it only appears when running a large number of videos.) and the problem still occurs, could you please share the root cause?
If the issue does not occur on 8.0, could you provide some notes on the related release changes or fixes in 8.0?

I am using the default settings and have not provided a custom config file for newstreammux.
I noticed something like this, but I haven’t fine-tuned it yet, so I’m not sure whether it’s related.


[property]
algorithm-type=1
#batch-size=4
#max-fps-control disables (=0) and enables (=1)
#throttling of buffers in muxer to roughly achieve
#confligured max-fps setting
max-fps-control=1
overall-max-fps-n=30
overall-max-fps-d=1
overall-min-fps-n=25
overall-min-fps-d=1
max-same-source-frames=1

[source-config-0]
max-num-frames-per-batch=1

I have already attached the video files and the related configuration files above. parallel_issues.zip

1 Like
5

I’m trying to reproduce it; I’ll come back if I find anything.

6

I have partially reproduced your problem. Several factors contribute to this issue:

  1. Different branches of inference run at different speeds.
  2. The impact of the pts-tolerance parameter on merging meta tags in nvdsmetamux.
  3. The strategy for forming batches in nvstreammux.

When 5-th frame of source 0 enters batch-6 (this batch will be sent for inference), the 6-th frame enters batch-7.
However, due to the different preformance of the two branches and a large value of pts-tolerance for nvdsmetamux. The metadata output by branch 0 is merged into batch-6, and the metadata output by branch 1 is merged into batch-7. This is where the problem occurs.

5090/native code is faster and less reproduces to this issue; this issue may easy occur under high loading/Python code.

with the new nstreammux
pts-tolerance=20000 → work fine

batch count 0 source id 0 frame id 0 == objects
batch count 0 source id 1 frame id 0 == objects
batch count 0 source id 2 frame id 0 == objects
batch count 0 source id 3 frame id 0 == objects
batch count 1 source id 0 frame id 1 == null object
batch count 1 source id 1 frame id 1 == null object
batch count 1 source id 2 frame id 1 == null object
batch count 1 source id 3 frame id 1 == null object

pts-tolerance=60000 → only source 0/2 enter batch 0.

batch count 0 source id 0 frame id 0 == objects
batch count 0 source id 2 frame id 0 == objects
batch count 1 source id 0 frame id 1 == null object
batch count 1 source id 1 frame id 0 == objects
batch count 1 source id 2 frame id 1 == null object
batch count 1 source id 3 frame id 0 == objects

Here is my test case. I am using the latest deepstream-8.0 and latest deepstream_reference_apps/deepstream_parallel_inference_app and vehicle0_lpr_analytic/source4_1080p_dec_parallel_infer.yml configuration file.
I made the following modifications to the code.

diff --git a/deepstream_parallel_inference_app/tritonclient/sample/apps/deepstream-parallel-infer/deepstream_parallel_infer_app.cpp b/deepstream_parallel_inference_app/tritonclient/sample/apps/deepstream-parallel-infer/deepstream_parallel_infer_app.cpp
index 2e938ba..397232b 100755
--- a/deepstream_parallel_inference_app/tritonclient/sample/apps/deepstream-parallel-infer/deepstream_parallel_infer_app.cpp
+++ b/deepstream_parallel_inference_app/tritonclient/sample/apps/deepstream-parallel-infer/deepstream_parallel_infer_app.cpp
@@ -273,6 +273,7 @@ create_display_meta(Vec2D<int> &objects, Vec3D<float> &normalized_peaks, NvDsFra
   }
 }
 
+static int batch = 0;
 /* body_pose_gie_src_pad_buffer_probe  will extract metadata received from pgie
  * and update params for drawing rectangle, object information etc. */
 static GstPadProbeReturn
@@ -291,8 +292,8 @@ body_pose_gie_src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
   {
     NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
 
-    if (frame_meta->batch_id == 0)
-      g_print("Processing frame number = %d\t\n", frame_meta->frame_num);
+    // if (frame_meta->batch_id == 0)
+    //   g_print("Processing frame number = %d\t\n", frame_meta->frame_num);
 
     for (l_user = frame_meta->frame_user_meta_list; l_user != NULL;
          l_user = l_user->next)
@@ -309,10 +310,13 @@ body_pose_gie_src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
       }
     }
 
+
+    printf("batch count %d source id %d frame id %d == %s\n", batch, frame_meta->source_id, frame_meta->frame_num, (frame_meta->obj_meta_list == nullptr) ? "null object" : "objects");
     for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
          l_obj = l_obj->next)
     {
       NvDsObjectMeta *obj_meta = (NvDsObjectMeta *)l_obj->data;
+
       for (l_user = obj_meta->obj_user_meta_list; l_user != NULL;
            l_user = l_user->next)
       {
@@ -329,6 +333,7 @@ body_pose_gie_src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
       }
     }
   }
+  batch++;
   return GST_PAD_PROBE_OK;
 }

sources_4_different_source.csv（source 0/1 30fps, source 2/3 is 25fps)

1,3,file:///root/deepstream_reference_apps/deepstream_parallel_inference_app/traffic.mp4,1,0,0
1,3,file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4,1,0,0
1,3,file:///root/deepstream_reference_apps/deepstream_parallel_inference_app/output.mp4,1,0,0
1,3,file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_qHD.mp4,1,0,0

The legacy nvstreammux is getting stuck; this is a bug, but I think this issue will occur in both legacy and new streammux.
This problem is unavoidable; the only solution is to adjust the parameters to prevent it from occurring. This is actually the expected behavior.

7

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.