How to convert NvBufSurface to cv::Mat

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello, Could you refer me to some sample implementation for converting the NvBufSurface to std::vector of cv::Mat. The deepstream-appsrc-test does not access the buffer it only access the metadata. I have tried this code to access the buffer and I can access it but I can not convert it.

NvBufSurface *surface = NULL;
memset(&in_map_info, 0, sizeof(in_map_info));
if (!gst_buffer_map(buf, &in_map_info, GST_MAP_READ))
{
  g_print("Error: Failed to map gst buffer\n");
}
surface = (NvBufSurface *)in_map_info.data;
1 Like
3

Hello I have used this code to convert the buffer to cv::Mat but when I show the image with cv::imshow it does not show the image it show a black screen and the window that opencv imshow creates its’s size is smaller than the image’s dimensions. g_imagesBatch is a global std::vector<cv::Mat> and all of this is happening inside appsink callback.

for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
         l_frame = l_frame->next)
    {
      std::cout << "Number of frames meta data: " << batch_meta->num_frames_in_batch << std::endl;
      NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);

      if (surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0] == NULL)
      {
        if (NvBufSurfaceMap(surface, frame_meta->batch_id, 0, NVBUF_MAP_READ_WRITE) != 0)
        {
          std::cout << "Faild to map the surface buffer\n";
          return GST_FLOW_ERROR;
        }
      }

      if (surface->memType == NVBUF_MEM_SURFACE_ARRAY)
      {
        NvBufSurfaceSyncForCpu(surface, frame_meta->batch_id, 0);
      }

      cv::Mat in_mat;

      in_mat =
            cv::Mat (surface->surfaceList[frame_meta->batch_id].planeParams.height[0],
            surface->surfaceList[frame_meta->batch_id].planeParams.width[0], CV_8UC4,
            surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
            surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);

      g_imagesBatch.push_back(in_mat);

      if (NvBufSurfaceUnMap (surface, frame_meta->batch_id, 0))
      {
        std::cout<<"Failed to unmap surface buffer\n";
      }

}
4

Hello @ahmed.t.mohamed

Maybe you can use

surface->surfaceList[frame_meta->batch_id].pitch

instead of

surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]

Source: DeepStream get cv::mat wrong

Hope it works
Regards

5

Thanks for your reply I changed it and it still don’t work the frame is black and small.

In my pipeline I set the nvvideoconverter memory type 3 I tried all the types but nothing works.
g_object_set(G_OBJECT(nvvidconv), "nvbuf-memory-type", 3, NULL);

Screenshot from 2021-05-20 16-47-01

6

Could you refer Deepstream sample code snippet - #3 by bcao

7

Thanks this was very helpful but I have some questions this is my pipeline:

decodebin->nvstreammux->nvinfer->nvtracker->nvvideoconverter->appsink

isn’t the nvvideoconverter is converting the Nv12 image to RGBA image ? and the appsink is after the nvvideoconverter image. The code in the example you referred me to is getting the image in a callback after nvinfer so it is receiving the image as Nv12 image and converting it to RGBA image and if I am receiving the image after the nvvideoconverter I don’t need to convert the surface buffer to cv::Mat of type Nv12 and then convert it to RGBA I need to convert the surface buffer to cv::Mat of type RGBA
and Is it wrong putting the appsink after the nvvideoconverter and is this code wrong for reading the surface buffer as cv::Mat of type RGBA in the appsink ?

Thanks in advance.

for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
         l_frame = l_frame->next)
    {
      std::cout << "Number of frames meta data: " << batch_meta->num_frames_in_batch << std::endl;
      NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);

      if (surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0] == NULL)
      {
        if (NvBufSurfaceMap(surface, frame_meta->batch_id, 0, NVBUF_MAP_READ_WRITE) != 0)
        {
          std::cout << "Faild to map the surface buffer\n";
          return GST_FLOW_ERROR;
        }
      }

      if (surface->memType == NVBUF_MEM_SURFACE_ARRAY)
      {
        NvBufSurfaceSyncForCpu(surface, frame_meta->batch_id, 0);
      }

      cv::Mat in_mat;

      in_mat =
            cv::Mat (surface->surfaceList[frame_meta->batch_id].planeParams.height[0],
            surface->surfaceList[frame_meta->batch_id].planeParams.width[0], CV_8UC4,
            surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
            surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);

      g_imagesBatch.push_back(in_mat);

      if (NvBufSurfaceUnMap (surface, frame_meta->batch_id, 0))
      {
        std::cout<<"Failed to unmap surface buffer\n";
      }

}
8

I guess here is unsafe since you had unMap the surface but you still can access the in_mat.