The problem of the above pipeline is there is no guarantee the frame 1 of left.h264 will follow the frame 1 of right.h264 for stereo plugin. Some times the left.h264 ahead of right.h264 by 4 frames. And some times ahead of 3 frames. There is almost never in sync even I add sync=True at the end of the pipeline.
Question: is there a way to guarantee the left.h264 and right.h264 in sync using nvstreammux? that is either left.h264 ahead by only 1 frame of right.h264 or vice versa. How? Please advise. Thanks a lot for your help.
Two filesrcs mean two separated sources. There is no mechanism in open source gstreamer to sync these two videos. nvstreammux is a common use plugin for inference preparing, it can not know the special relationship between these two videos.
Then is there a way to use, for example, python to read in images from both files and then feed to the pipeline? or other API I should use instead of Deepstream to feed the left/right images to nvinfer object detector to get bounding box result? Please advise. Thank you.
combine left frame and right frame into one frame input to the pipeline
read a left frame and read a right frame, then push a left frame into appsrc 1 and push a right frame into appsrc 2, such that the app gate the files read and buffer push to guarantee the sync
For the current software already developed using deepstream we have, we prefer use option 2.
For two appsrc plugins, they are still two separated streams. It is just the same case with two filesrc plugins. Maybe you can try to attached same timestamp to the frames which you think should be together with your appsrc implementation. But this is still not guaranteed.