How to emulate a single source as multiple sources for nvmultistreamtiler

• Hardware Platform (Jetson / GPU) 1080Ti/AGX Xavier
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4
• NVIDIA GPU Driver Version (valid for GPU only) 450.80.02

I have a “my-plugin” developed and run the following pipeline well:

gst-launch-1.0 filesrc location= /data/agx/al.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 !
nvinfer config-file-path= dstest1_pgie_config.txt ! nvvideoconvert !
my_plugin ! nvdsosd ! nveglglessink sync=False

Now I would like to display the even frame as source-id=0 and odd frame as source-id=1 using nvmultistreamtiler as below:

gst-launch-1.0 filesrc location= /data/agx/al.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 !
nvinfer config-file-path= dstest1_pgie_config.txt ! nvvideoconvert !
my_plugin ! nvmultistreamtiler ! nvdsosd ! nveglglessink sync=False

So what I did in “my_plugin” is to modify the frame_meta->source_id as below:

frame_meta->source_id = frame_meta->frame_num % 2; //0:left, 1:right

When I run it, it seems the nvmultistreamtiler still see only one source and not tile of left/right images.

Question: is frame_meta->source_id the right variable to change to make nvmultistreamtiler working? or other variable?

Thank you for your help.

Most deepstream plugin works on batch. Your batch size is always 1 with this pipeline. So nvmultistreamtiler does not think there are two frames in one batch.

You can try to split the video into two streams before nvstreammux.

The following two streams pipeline also only has batch-size=1, but the nvmultistreamtiler works as expected, it seems not thing to do with batch-size, why?

gst-launch-1.0
filesrc location= left.h264 ! h264parse ! nvv4l2decoder name=c102
filesrc location= right.h264 ! h264parse ! nvv4l2decoder name=c104
c102. ! nvvideoconvert nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080
c104. ! nvvideoconvert nvbuf-memory-type=3 ! m.sink_1
m. ! nvinfer config-file-path= dstest1_pgie_config.txt !
nvvideoconvert ! nvmultistreamtiler width=2560 height=720 ! nvdsosd ! nveglglessink

Please set “batch-size=2” for your nvstreammux plugin and set “rows=1 columns=2” for your nvmultistreamtiler plugin.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux.html
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmultistreamtiler.html

Thank you @Fiona.Chen for your help.

It turns out that nvmultistreamtiler depends on frame_meta->pad_index to tile the frames.

So instead of only modify frame_meta->source_id, I did the following:

frame_meta->pad_index=frame_meta->source_id = frame_meta->frame_num % 2; //0:left, 1:right

And now nvmultistreamtiler works as expected!

Thank you.