# Save frames extracted from Deepstream pipeline in C++ OpenCV

**URL:** <https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389>\
**Category:** DeepStream SDK\
**Created:** [October 23, 2023, 4:27pm UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389 "2023-10-23T16:27:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![StrikeEagleIII](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/strikeeagleiii/32/202096_2.png) [@StrikeEagleIII](https://forums.developer.nvidia.com/u/StrikeEagleIII)\
**Post date:** [October 23, 2023, 4:27pm UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/1 "2023-10-23T16:27:43Z")

</div>

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU)**: Jetson Xavier AGX / NX, Orin AGX / NX  
**• DeepStream Version** 6.0.1  
**• JetPack Version (valid for Jetson only)** 4.6.3  
**• Issue Type( questions, new requirements, bugs)** question

I’m trying to extract image frames from the metadata (similar to [here](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-to-video-using-opencv/216478) except in C++ instead of python) and instead of saving them to disk, send them to my own custom gui.

Basically, in the probe callback from nvinfer, I would like to grab the image so I can do other things with it. The `deepstream_image_meta_test example.c` [this post](https://forums.developer.nvidia.com/t/save-frames-in-deepstream-sample-test-1-c-code/249742/5) points to is close, however in the example, it’s just sends the `ip_surf` pointer to the function `nvds_obj_enc_process` which does all of the magic of extracting the image data, encoding it to jpeg and saving it. I need to know what that function does.

`ip_surf` is a `NvBufSurface` which according to documentation holds information about batched buffers. It looks like there could be a couple of places where the data might be:

```auto
ip_surf->surfaceList (NvBufSurfaceParams) -> dataPtr
ip_surf->surfaceList ->mappedAddr (NvBufSurfaceMappedAddr)->eglImage
ip_surf->surfaceList ->mappedAddr->addr

```

however both `eglImage `and `addr` are null pointers. `dataPtr` has something, but when I converted to BGR and save as a jpg (as a test) I just get garbage:

 ![frame0](https://global.discourse-cdn.com/nvidia/original/4X/f/8/5/f850a85b3c2d6d6565a28e9aa257cdf7f7d70790.jpeg)  
Code is:

```auto
 GstMapInfo inmap = GST_MAP_INFO_INIT;
 gst_buffer_map(buf, &inmap, GST_MAP_READ)
 NvBufSurface* ip_surf = (NvBufSurface*)inmap.data;
 gst_buffer_unmap(buf, &inmap);
 NvBufSurfaceParams* surf = ip_surf->surfaceList;
 cv::Mat image {(int)surf->height * 3 / 2, (int)surf->width, CV_8UC1, surf->dataPtr};
 cv::Mat image_bgr;
 cv::cvtColor(image, image_bgr, cv::COLOR_YUV2BGR_NV12);
 
cv::imwrite(fmt::format("frame{}.jpg", frame_meta->frame_num), image_bgr);

```

Any help would be appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![yuweiw](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yuweiw/32/101255_2.png) [@yuweiw](https://forums.developer.nvidia.com/u/yuweiw)\
**Post date:** [October 24, 2023, 3:15am UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/3 "2023-10-24T03:15:44Z")

</div>

You can refer to our source code: `opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-dsexample\gstdsexample_optimized.cpp`. There is a similar usage in the code.

```auto
static gboolean
convert_batch_and_push_to_process_thread (GstDsExample * dsexample,
    GstDsExampleBatch * batch) {
......
#ifdef WITH_OPENCV
    in_mat =
        cv::Mat (dsexample->processing_height, dsexample->processing_width,
        CV_8UC4,dsexample->inter_buf->surfaceList[i].mappedAddr.addr[0],
      dsexample->inter_buf->surfaceList[i].pitch);
......

}

```

---

<div class="post-metadata">

**Author:** ![StrikeEagleIII](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/strikeeagleiii/32/202096_2.png) [@StrikeEagleIII](https://forums.developer.nvidia.com/u/StrikeEagleIII)\
**Post date:** [October 24, 2023, 9:49pm UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/4 "2023-10-24T21:49:24Z")

</div>

I’m looking at that and it’s not clear what prep work is needed for that call to succeed. Looks like the steps are(loops and error checking removing for clarity)

```auto
NvBufSurfaceMemSet (dsexample->inter_buf, i, 0, 0);
err = NvBufSurfTransform (&dsexample->batch_insurf, dsexample->inter_buf,
      &dsexample->transform_params)
nvtxDomainRangePop (dsexample->nvtx_domain); // is this needed?
NvBufSurfaceMap (dsexample->inter_buf, i, 0, NVBUF_MAP_READ)
NvBufSurfaceSyncForCpu (dsexample->inter_buf, i,0);

```

Give me a day or two to work through that. Thanks,

---

<div class="post-metadata">

**Author:** ![yuweiw](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yuweiw/32/101255_2.png) [@yuweiw](https://forums.developer.nvidia.com/u/yuweiw)\
**Post date:** [October 25, 2023, 6:07am UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/5 "2023-10-25T06:07:27Z")

</div>

OK. `nvtxDomainRangePushEx` and `nvtxDomainRangePop` are just for debug. You can ignore that. If you want to debug that plugin, you can use the following demo. `opt\nvidia\deepstream\deepstream\sources\apps\sample_apps\deepstream-opencv-test`.

---

<div class="post-metadata">

**Author:** ![StrikeEagleIII](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/strikeeagleiii/32/202096_2.png) [@StrikeEagleIII](https://forums.developer.nvidia.com/u/StrikeEagleIII)\
**Post date:** [October 26, 2023, 2:30pm UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/6 "2023-10-26T14:30:42Z")

</div>

So here’s what I have that appears to work:

```auto
tl::expected<cv::Mat, GstFlowReturn> get_opencv_mat(NvBufSurface& ip_surf, gint idx,
                                                      gdouble& ratio, gint processing_width,
                                                      gint processing_height) {
    // Not sure we actually need to copy the NvBufSurface arg above (originally named input_buf), so
    // instead of making a copy make it a reference and change the name to ip_surf so it can be used
    // directly? 
    
    // then we don't need
    // NvBufSurface ip_surf; 
    // ip_surf = *input_buf; 
    // ip_surf.numFilled = ip_surf.batchSize = 1; 
    // ip_surf.surfaceList = &(input_buf->surfaceList[idx]);

    // Where we're grabbing the entire image, I'm not sure any of this scaling stuff is needed from
    // here to the NvBufSurfTransformParams definition
    NvOSD_RectParams rect_params;
    // Scale the entire frame to processing resolution
    rect_params.left = 0;
    rect_params.top = 0;
    rect_params.width = processing_width;
    rect_params.height = processing_height;

    gint src_left = GST_ROUND_UP_2((int)rect_params.left);
    gint src_top = GST_ROUND_UP_2((int)rect_params.top);
    gint src_width = GST_ROUND_DOWN_2((int)rect_params.width);
    gint src_height = GST_ROUND_DOWN_2((int)rect_params.height);

    // Maintain aspect ratio
    double hdest = processing_width * src_height / (double)src_width;
    double wdest = processing_height * src_width / (double)src_height;
    guint dest_width, dest_height;

    if (hdest <= processing_height) {
      dest_width = processing_width;
      dest_height = hdest;
    } else {
      dest_width = wdest;
      dest_height = processing_height;
    }

    // Configure transform session parameters for the transformation
    NvBufSurfTransformConfigParams transform_config_params;
    transform_config_params.compute_mode = NvBufSurfTransformCompute_Default;
    transform_config_params.gpu_id = gpu_id_;
    transform_config_params.cuda_stream = cuda_stream_;

    // Set the transform session parameters for the conversions executed in this thread.
    if (auto err = NvBufSurfTransformSetSessionParams(&transform_config_params);
        err != NvBufSurfTransformError_Success) {
      spdlog::error("NvBufSurfTransformSetSessionParams failed with error {}", err);
      return tl::unexpected {GST_FLOW_ERROR};
    }

    // Calculate scaling ratio while maintaining aspect ratio
    ratio = MIN(1.0 * dest_width / src_width, 1.0 * dest_height / src_height);

    if ((rect_params.width == 0) || (rect_params.height == 0)) {
      spdlog::error("[get_opencv_mat]:crop_rect_params dimensions are zero");
      return tl::unexpected {GST_FLOW_ERROR};
    }

#ifdef __aarch64__
    if (ratio <= 1.0 / 16 || ratio >= 16.0) {
      // Currently cannot scale by ratio > 16 or < 1/16 for Jetson
      spdlog::error("[get_opencv_mat] ratio {} not in range of [.0625 : 16]", ratio);
      return tl::unexpected {GST_FLOW_ERROR};
    }
#endif
    // Set the transform ROIs for source and destination
    NvBufSurfTransformRect src_rect = {(guint)src_top, (guint)src_left, (guint)src_width,
                                       (guint)src_height};
    NvBufSurfTransformRect dst_rect = {0, 0, (guint)dest_width, (guint)dest_height};

    // Set the transform parameters
    NvBufSurfTransformParams transform_params;
    transform_params.src_rect = &src_rect;
    transform_params.dst_rect = &dst_rect;
    transform_params.transform_flag =
      NVBUFSURF_TRANSFORM_FILTER | NVBUFSURF_TRANSFORM_CROP_SRC | NVBUFSURF_TRANSFORM_CROP_DST;
    transform_params.transform_filter = NvBufSurfTransformInter_Default;

    // Memset the memory
    NvBufSurfaceMemSet(inter_buf_, idx, 0, 0);

    // spdlog::debug("Scaling and converting input buffer");

    // Transformation scaling+format conversion if any. Not sure if the transform is actually doing
    // anything other than a copy? Maybe converting from NV12 to RGBA? Can we convert from NV12 to
    // BGR directly so only one conversion is needed?
    if (auto err = NvBufSurfTransform(&ip_surf, inter_buf_, &transform_params);
        err != NvBufSurfTransformError_Success) {
      spdlog::error("NvBufSurfTransform failed with error {} while converting buffer", err);
      return tl::unexpected {GST_FLOW_ERROR};
    }
    // Map the buffer so that it can be accessed by CPU
    if (NvBufSurfaceMap(inter_buf_, idx, 0, NVBUF_MAP_READ) != 0) {
      return tl::unexpected {GST_FLOW_ERROR};
    }

    // Cache the mapped data for CPU access
    NvBufSurfaceSyncForCpu(inter_buf_, idx, 0);

    // Use openCV to remove padding and convert RGBA to BGR. Can be skipped if
    // algorithm can handle padded RGBA data.
    const auto in_mat =
      cv::Mat(processing_height, processing_width, CV_8UC4,
              inter_buf_->surfaceList[0].mappedAddr.addr[0], inter_buf_->surfaceList[0].pitch);

    cv::Mat image_bgr;
    cv::cvtColor(in_mat, image_bgr, cv::COLOR_RGBA2BGR);

    if (NvBufSurfaceUnMap(inter_buf_, idx, 0)) {
      return tl::unexpected {GST_FLOW_ERROR};
    }

    return image_bgr;
  }

```

which is called in the src pad probe callback from nvinfer:

```auto
  static GstPadProbeReturn pgie_src_pad_buffer_probe(GstPad* pad, GstPadProbeInfo* info,
                                                     gpointer u_data) {

    GstCaps* caps = gst_pad_get_current_caps(pad);
    GstVideoInfo video_info = {};
    if (!gst_video_info_from_caps(&video_info, caps)) {
      logger_->error("[ProcessFrame] failed to get video_info");
      return (GstPadProbeReturn)GST_FLOW_ERROR;
    }

    gst_caps_unref(caps);

    // NvDsDisplayMeta *display_meta = NULL;
    
    GstBuffer* buf = (GstBuffer*)info->data;

    GstMapInfo inmap = GST_MAP_INFO_INIT;
    if (!gst_buffer_map(buf, &inmap, GST_MAP_READ)) {
      GST_ERROR("input buffer mapinfo failed");
      return (GstPadProbeReturn)GST_FLOW_ERROR;
    }
    if (inter_buf_ == nullptr) {
      logger_->info("[ProcessFrame] initializing Frame buffer");
      NvBufSurfaceCreateParams create_params;
      create_params.gpuId = gpu_id_;
      create_params.width = video_info.width; // dsexample->processing_width;
      create_params.height = video_info.height; // dsexample->processing_height;
      create_params.size = 0;
      create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
      create_params.layout = NVBUF_LAYOUT_PITCH;
#ifdef __aarch64__
      create_params.memType = NVBUF_MEM_DEFAULT;
#else
      create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
#endif
      if (NvBufSurfaceCreate(&inter_buf_, 1, &create_params) != 0) {
        logger_->error("Error: Could not allocate internal buffer for dsexample");
        GST_ERROR("Error: Could not allocate internal buffer for dsexample");
        return (GstPadProbeReturn)GST_FLOW_ERROR;
      }
    }
    NvBufSurface* ip_surf = (NvBufSurface*)inmap.data;
    gst_buffer_unmap(buf, &inmap);

    const int index = 0; // at the moment our batch size == 1 so to simplify things, start there
                          // instead of looping through buffers, etc.
    double scale_ratio = 1.0;
    const auto frame_width = ip_surf->surfaceList[0].width;
    const auto frame_height = ip_surf->surfaceList[0].height;
    cv::Mat frame;
    if (const auto ret =
          vp->get_opencv_mat(*ip_surf, index, scale_ratio, frame_width, frame_height);
        !ret.has_value()) {
      return (GstPadProbeReturn)ret.error();
    } else {
      frame = ret.value();
    }

```

If you have any suggestions for simplifying this (e.g. convert directly from nv12 to bgr) that would be great.

---

<div class="post-metadata">

**Author:** ![yuweiw](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yuweiw/32/101255_2.png) [@yuweiw](https://forums.developer.nvidia.com/u/yuweiw)\
**Post date:** [October 27, 2023, 2:28am UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/7 "2023-10-27T02:28:50Z")

</div>

Basically, if you just want to save images, you can refer to the source code you attached:`deepstream_image_meta_test.c`.  
If you just want to convert the colorformat, you can just use our [NvBufSurfTransform](https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/group __ds__ bbb.html#gab8e0c12fdf8f3e0215a17765b2eed4eb) API.  
You can also use our `nvvideoconvert` plugin to convert that directly.

---

<div class="post-metadata">

**Author:** ![StrikeEagleIII](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/strikeeagleiii/32/202096_2.png) [@StrikeEagleIII](https://forums.developer.nvidia.com/u/StrikeEagleIII)\
**Post date:** [October 27, 2023, 2:27pm UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/8 "2023-10-27T14:27:45Z")

</div>

I’m not just interested in saving images, that was just the easiest thing to do for a proof of concept. It would be nice if the `nvds_obj_enc_process` function from that example was broken up so you could just pick the part of it you need (i.e. get the cv::Mat object).

Interesting comment about the `nvvideoconvert`. I do have that element in the pipeline. I would assume the callback we have now on the nvinfer element src pad could be moved to the nvvideoconvert src pad and I would assume I would still have access to all the metadata (i.e. detection bounding boxes, classes, etc.) as NVOSD is still downstream of `nvvideoconvert` and needs that metadata

---

<div class="post-metadata">

**Author:** ![yuweiw](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yuweiw/32/101255_2.png) [@yuweiw](https://forums.developer.nvidia.com/u/yuweiw)\
**Post date:** [October 30, 2023, 1:56am UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/9 "2023-10-30T01:56:37Z")

</div>

**There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks**

We currently have no plans to open source `nvds_obj_enc_process`, but we have provided multiple parameter configurations to configure the output mode. You can refer to the soure code `NvDsObjEncUsrArgs` in `nvds_obj_encode.h`.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [November 13, 2023, 9:28am UTC](https://forums.developer.nvidia.com/t/save-frames-extracted-from-deepstream-pipeline-in-c-opencv/270389/10 "2023-11-13T09:28:41Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.