Combining original frame with inference results

[vitalii.vovk7]

Hi,
I have such kind of pipeline:
nvurisrcbin → custompreprocess → nvstreammux → nvinfer → nvstreamdemux → custompostprocessing → fakesink

Where custompreprocess modifies the original frame (let it be ROI cropping).
The problem is I want to have the original frame and inference results in custompostprocessing.

How could I achieve this?

I thought about using tee and nvstreammux plugins but the pipeline freezes after a few frames. Here is example:

        nvurisrcbin uri=<some_rtsp> rtsp-reconnect-interval=30 !
        tee name=t
      
        t. ! 
        queue ! 
        cmux.sink_0
        
        t. ! 
        mux.sink_0

        nvstreammux name=mux height=640 width=640 batched-push-timeout=40000 batch-size=1 nvbuf-memory-type=3 live-source=1 !
        gstcustompreprocess !
        nvinfer config-file-path=<conf_pth> model-engine-file=<engine_pth> !
        nvstreamdemux name=demux
        
        demux.src_0 ! 
        nvvideoconvert nvbuf-memory-type=3  ! 
        video/x-raw(memory:NVMM), format=RGBA !
        queue ! 
        cmux.sink_1

        nvstreammux name=cmux sync-inputs=1 max-latency=12500000 batched-push-timeout=40000 height=640 width=640 batch-size=2 nvbuf-memory-type=3 live-source=1 ! 
        gstcustompostprocess !
        nvstreamdemux name=cdemux
        cdemux.src_0 !
        fakesink async=0

[junshengy]

Sorry for the long delay.

I think you can refer to the nvdspostprocess plugin. nvdpostprocess can process batches, so there is no need to add nvstreamdemux upstream

Similarly, nvdspreprocess needs to be placed downstream of nvstreammux

So the pipeline like the below

nvurisrcbin --> nvstreammux --> nvdspreprocess(with user library) --> nvinfer --> nvdspostprocess(with user library) --> fakesink

It’s open source, /opt/nvidia/deepstream/deepstream-7.0/sources/gst-plugins/gst-nvdspostprocess.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdspostprocess.html