How to convert from ds3d::datamap to NvBufSurface

I saw that one of DeepStream v7 features is the ds3d::databridge plugin, which allows the conversion from NvBufSurface to ds3d::datamap. Now, what if I want to do the opposite? e.g.: extract an image encoded in a datamap using ds3d::Frame2DGuard and convert it to a NvBufSurface.

Currently there is no ds3d 3d to 2d bridge filter. Can you tell us why do you need such conversion? If the data is video data(RGB, I420, YUYV, UYVY,…), you can use NvBufSurface to get it directly. Where and how did you get the ds3d datamap data?

Can you tell me more about retrieving video data using NvBufSurface?

I want this conversion so I can mix 2D and 3D elements in my pipeline. So far, from what I understand, once you have a datamap buffer, you cannot use DeepStream 2D plugins with it.

A simple scenario using this mix of 2D and 3D plugins would be:

                                 ┌──────────┐  ┌─────────────────┐  ┌────────────────┐                                       
                               ┌─┤gst::queue├──┤nvgeneric2dplugin├──┤ds3d::databridge├─┐                                     
                               │ └──────────┘  └─────────────────┘  └────────────────┘ │                                     
┌────────────────┐  ┌────────┐ │                                                       │ ┌───────────────┐ ┌────────────────┐
│ds3d::dataloader├──┤gst::tee├─┤                                                       ├─┤ds3d::datamixer├─┤ds3d::datarender│
└────────────────┘  └────────┘ │                                                       │ └───────────────┘ └────────────────┘
                               │ ┌──────────┐  ┌────────────────┐                      │                                     
                               └─┤gst::queue├──┤ds3d::datafilter├──────────────────────┘                                     
                                 └──────────┘  └────────────────┘                                                            

What is your video source? Video files such as mp4, mkv, …? Live video streams such as RTSP, http,…? Camera device such as CSI, USB,…? Or others?

The pipeline is not scenario. Can you tell us your real scenario? What kind of data do you want to process? How does the data be involved to the pipeline(read from data file, from special devices, from network streams,…)?

My video source comes from the RealSense SDK.

My scenario is as follows: I read data from RealSense (e.g., color image, depth map, calibration matrices) using ds3d::dataloader. Then I use a ds3d::datafilter to perform object detection on the color image and use the depth map to get the distance of each object. That said, instead of writing my custom ds3d::datafilter to handle the object detection, I would like to convert the color image from ds3d::datamap to NvBufSurface, so I can use plugins like nvinfer or nvdsosd for this process. In the end, ideally, the pipeline would look something like this:

                                                         (3D to 2D)                                               (2D to 3D)                                                                                         
                                        ┌──────────┐  ┌────────────────┐ video/x-raw ┌──────────┐  ┌───────┐  ┌────────────────┐    datamap                                                                          
                                ┌───────┤gst::queue├──┤ds3d::databridge├─────────────┤nvstreamux├──┤nvinfer├──┤ds3d::databridge├────────────────────────┐                                                            
                                │       └──────────┘  └────────────────┘ NvBufSurface└──────────┘  └───────┘  └────────────────┘     color              │                                                            
┌────────────────┐ datamap ┌────┴───┐                                                                                            NvDsBatchMeta  ┌───────┴───────┐    datamap    ┌────────────────┐ ┌────────────────┐
│ds3d::dataloader├─────────┤gst::tee│                                                                                                           │ds3d::datamixer├───────────────┤ds3d::datafilter├─┤ds3d::datarender│
└────────────────┘  color  └────┬───┘                                                                                                           └───────┬───────┘     color     └────────────────┘ └────────────────┘
                    depth       │       ┌──────────┐                                                                                                    │             depth                                          
                    calib       └───────┤gst::queue├────────────────────────────────────────────────────────────────────────────────────────────────────┘             calib                                          
                                        └──────────┘                                                                                                              NvDsBatchMeta