Tee and nvstreamdemux issue

[Osama100000]

---

• Hardware Platform (GPU):
• DeepStream Version: 6.3
• NVIDIA GPU Driver Version: 535.183.06
• Issue Type: Questions

---

DeepStream Pipeline Issue: Displaying Side-by-Side RGBA and OSD Streams

Hello,

I am working on a DeepStream pipeline where I need to display each stream’s output in separate pop-up windows. My pipeline structure is as follows:

[some plugins] -> tee -> ofvisual -> 
                      -> osd ->

Pipeline Overview:

• Some Plugins: These plugins output batched streams.
• ofvisual: This plugin converts the attached motion vectors to RGBA frames for each stream.
• nvdsosd: This plugin draws bounding boxes for each stream.

Objective:

I aim to create a pop-up window for each stream, split into two sections:

1. Left Side: Displays the RGBA frames generated by ofvisual.
2. Right Side: Displays the OSD frames with bounding boxes.

The goal is to have these two outputs tiled side by side for each stream in the same window.

Request:

I’m seeking guidance on implementing this in DeepStream. Specifically, I would appreciate advice on:

• Best practices for tiling the RGBA and OSD outputs side by side within the same window.

Any suggestions or examples would be greatly appreciated!

---

Approaches Tried:

1. Pipeline with nvstreamdemux and nvmultistreamtiler

some plugins -> tee -> queue -> nvofvisual -> nvstreamdemux (src_0) -> nvstreammux -> nvmultistreamtiler
                    -> queue -> nvdsosd ----> nvstreamdemux (src_0)

Issue: This approach results in synchronization problems. Only one part of the output (either flow motion or OSD) is shown, depending on the order of linkage in the code. I attached a probe on the sink pads of each nvstreamdemux to confirm the synchronization issue, and only one branch receives a buffer, as indicated by the probe output.

---

2. Pipeline with Tiler Configuration

nvstreamdemux ----(stream_0)-----> tee -> queue -> ofvisual -> nvstreammux -> tiler -> bins
                                       -> queue -> nvdsosd ->
    
nvstreamdemux ----(stream_1)-----> tee -> queue -> ofvisual -> nvstreammux -> tiler -> bins
                                       -> queue -> nvdsosd ->

Issue: This approach works for a single stream, but when adding another stream, the nvmultistreamtiler reconfigures to a (1x3) layout upon receiving an event for the added stream, even if it was pre-configured. Attempting to reconfigure it at runtime has no effect.

Note: All nvmultistreamtiler instances were configured to (1x2).

---

3. Pipeline with Compositor

nvstreamdemux ----(stream_0)-----> tee -> queue -> ofvisual -> compositor -> some sink bins
                                       -> queue -> nvdsosd ->
    
nvstreamdemux ----(stream_1)-----> tee -> queue -> ofvisual -> compositor -> some sink bins
                                       -> queue -> nvdsosd ->

Issue: This approach works fine for individual streams, but adding multiple streams leads to a significant drop in FPS, indicating a bottleneck. I also tried replacing the compositor with glvideomixer, but the FPS drop persisted with multiple streams.

---

Any insights or solutions to these issues would be greatly appreciated!

---

[yuweiw]

Do you have a fixed number of video sources or they are changed dynamically?

[Osama100000]

I am using nvmultiurisrcbin to dynamically add and remove sources, with the max_number_sources parameter defined to control the branching mechanism. For example, when max_number_sources is set to 2, it creates branches for stream 0 and stream 1.

[yuweiw]

Does the pop-up window means even if there are multiple sources, only one source of your selections could be displayed on the window?

[Osama100000]

Each source has its own sink bins, so the pop-up window is just render sink. In fact, the pop-up window is not required—it could be an RTSP stream or another output instead.

[yuweiw]

OK. If you use compositor plugin, it uses the software to process the video. This should be the cause of the bottleneck.

About the 1st pipeline, could you try to add 2 nvvideoconvert after the tee plugin and set the disable-passthrough to 1.

About the 2nd pipeline, theoretically one source corresponds to one tiler. So the tiler should not be reconfigured. Could you attach a simplified demo code to reproduce that?

[Osama100000]

Sorry for the late response
For The first Approach
I have been trying with your first suggestion on 1st pipeline by adding two nvvideoconverts and setting disable-passthrough to 1, which shows:

0:00:05.203161284 1484608 0x555556604580 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<sink_sub_bin_encoder1:pool:src> Uncertain or not enough buffers, enabling copy threshold

0:00:05.517646012 1484608 0x555556604700 WARN              nvofvisual gstnvofvisual.cpp:746:gst_nvof_visual_transform_internal:<nvopticalflowvisual> OF METADATA NOT FOUND

nvbufsurftransform_copy.cpp:96: => Buffer size mismatch

nvbufsurftransform_copy.cpp:604: => Failed to copy

nvbufsurftransform.cpp:3586: => Copy Transformation Failed -1

For the second approach, I will work on creating a simplified demo code, which may take some time. Additionally, as you mentioned, since each source has its own sink bins, they shouldn’t need to be reconfigured. However, I’m concerned that using a streammux to batch the streams and then a demux to separate them into RTSP output bins might cause this issue.

[yuweiw]

OK. Let’s dig deeper when we have a simplified demo. Also you can use the sudo tegrastats command to check the loading first when you get a bottleneck issue.

[yingliu]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks