Opencv mat clone osd frame coredump

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson /)
**• DeepStream Version 6.3
**• JetPack Version (valid for Jetson only) 5.1.2
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

hello,
I want save frame with opencv at osd_sink_pad_buffer_probe(), application is deepstream-test1, this How to convert NvBufSurface to cv::Mat - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums is demo,but third time core dump,

Another question is frame color format, deepstream-test1 osd_sink_pad_buffer_probe frame color format is 6(NV12), but my application is 19 (RGBA),cv::mat clone not core dump, but memory very increase fast. I want know how to set it , it depend what?
code like this:


static GstPadProbeReturn
osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
    GstBuffer *buf = (GstBuffer *) info->data;
    guint num_rects = 0;
    NvDsObjectMeta *obj_meta = NULL;
    guint vehicle_count = 0;
    guint person_count = 0;
    NvDsMetaList * l_frame = NULL;
    NvDsMetaList * l_obj = NULL;
    NvDsDisplayMeta *display_meta = NULL;
    GstMapInfo mapInfo;
    memset(&mapInfo, 0, sizeof(mapInfo));
    if (!gst_buffer_map(buf, &mapInfo, GST_MAP_READ))
    {
        g_error("Error: Failed to map gst buffer\n");
    }
    NvBufSurface *surface = (NvBufSurface *)mapInfo.data;
    NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
        int offset = 0;
        if (NvBufSurfaceMap(surface, -1, -1, NVBUF_MAP_READ) != 0)
        {
          printf("buffer map to be accessed by CPU failed\n");
        }

        NvBufSurfaceSyncForCpu(surface, -1, -1);
        printf("infer w %d h %d color format %d \n", surface->surfaceList[frame_meta->batch_id].planeParams.width[0],
                 surface->surfaceList[frame_meta->batch_id].planeParams.height[0],
                 (int)(surface->surfaceList[frame_meta->batch_id].colorFormat));
        guint height = surface->surfaceList[frame_meta->batch_id].height;
        guint width = surface->surfaceList[frame_meta->batch_id].width;

        cv::Mat frameMat = cv::Mat(height*3/2,
                                   width, CV_8UC1,
                                   surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
                                   surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);
        cv::Mat frameOri = frameMat.clone();
        NvBufSurfaceUnMap(surface, -1, -1);

        for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
                l_obj = l_obj->next) {
            obj_meta = (NvDsObjectMeta *) (l_obj->data);
            if (obj_meta->class_id == PGIE_CLASS_ID_VEHICLE) {
                vehicle_count++;
                num_rects++;
            }
            if (obj_meta->class_id == PGIE_CLASS_ID_PERSON) {
                person_count++;
                num_rects++;
            }
        }
        display_meta = nvds_acquire_display_meta_from_pool(batch_meta);
        NvOSD_TextParams *txt_params  = &display_meta->text_params[0];
        display_meta->num_labels = 1;
        txt_params->display_text = (char*)g_malloc0 (MAX_DISPLAY_LEN);
        offset = snprintf(txt_params->display_text, MAX_DISPLAY_LEN, "Person = %d ", person_count);
        offset = snprintf(txt_params->display_text + offset , MAX_DISPLAY_LEN, "Vehicle = %d ", vehicle_count);

        /* Now set the offsets where the string should appear */
        txt_params->x_offset = 10;
        txt_params->y_offset = 12;

        /* Font , font-color and font-size */
        txt_params->font_params.font_name = "Serif";
        txt_params->font_params.font_size = 10;
        txt_params->font_params.font_color.red = 1.0;
        txt_params->font_params.font_color.green = 1.0;
        txt_params->font_params.font_color.blue = 1.0;
        txt_params->font_params.font_color.alpha = 1.0;

        /* Text background color */
        txt_params->set_bg_clr = 1;
        txt_params->text_bg_clr.red = 0.0;
        txt_params->text_bg_clr.green = 0.0;
        txt_params->text_bg_clr.blue = 0.0;
        txt_params->text_bg_clr.alpha = 1.0;

        nvds_add_display_meta_to_frame(frame_meta, display_meta);
    }

    g_print ("Frame Number = %d Number of objects = %d "
            "Vehicle Count = %d Person Count = %d\n",
            frame_number, num_rects, vehicle_count, person_count);
    frame_number++;
    return GST_PAD_PROBE_OK;
}

We have a faster way to save pictures with nvds_obj_enc_process, you can refer to our demo.

sources\apps\sample_apps\deepstream-image-meta-test

@bcao please help me about this question

I read this demo, it encode frame to jpg and save to disk, but I need frame in host memory for another use(opencv). NvBufSurfaceMap this way is work well in deepstream6.0.

The cause may be a format mismatch. You can refer to our source code convert_batch_and_push_to_process_thread in the sources\gst-plugins\gst-dsexample\gstdsexample_optimized.cpp. First use the following API NvBufSurfTransform to convert the format to meet your needs.

@yuweiw this also my question about frame format, why format is different between deepstream-test1 and my application. My application format is RGBA , cv mat code:

        if (NvBufSurfaceMap(surface, -1, -1, NVBUF_MAP_READ) != 0)
        {
            LOG_ERROR("buffer map to be accessed by CPU failed\n");
        }

        NvBufSurfaceSyncForCpu(surface, -1, -1);
        LOG_INFO("infer w{} h{} color format {}",surface->surfaceList[frameMeta->batch_id].planeParams.width[0],
        surface->surfaceList[frameMeta->batch_id].planeParams.height[0],
        std::to_string(surface->surfaceList[frameMeta->batch_id].colorFormat));

        cv::Mat frameMat = cv::Mat(surface->surfaceList[frameMeta->batch_id].planeParams.height[0],
                                   surface->surfaceList[frameMeta->batch_id].planeParams.width[0], CV_8UC4,
                                   surface->surfaceList[frameMeta->batch_id].mappedAddr.addr[0],
                                   surface->surfaceList[frameMeta->batch_id].planeParams.pitch[0]);
        cv::Mat frameOri = frameMat.clone();

deepstream-test1 format is NV12,cv mat code:

if (NvBufSurfaceMap(surface, -1, -1, NVBUF_MAP_READ) != 0)
        {
          printf("buffer map to be accessed by CPU failed\n");
        }

        NvBufSurfaceSyncForCpu(surface, -1, -1);
        printf("infer w %d h %d color format %d \n", surface->surfaceList[frame_meta->batch_id].planeParams.width[0],
                 surface->surfaceList[frame_meta->batch_id].planeParams.height[0],
                 (int)(surface->surfaceList[frame_meta->batch_id].colorFormat));
        guint height = surface->surfaceList[frame_meta->batch_id].height;
        guint width = surface->surfaceList[frame_meta->batch_id].width;

        cv::Mat frameMat = cv::Mat(height*3/2,
                                   width, CV_8UC1,
                                   surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
                                   surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);
        cv::Mat frameOri = frameMat.clone();
        NvBufSurfaceUnMap(surface, -1, -1);

they all not work.

The format depends on the piepline you are using. For the deepstream-test1, the output format of the nvv4l2decoder plugin is NV12. For your application , it depends on your own pipeline.

Can you reproduce this question with opencv in deepstream-test1?

NV12 or RGBA is ok for me, because opencv can process them, the problem is how to copy frame to host memory correctly?

Please refer to our source code convert_batch_and_push_to_process_thread in the sources\gst-plugins\gst-dsexample\gstdsexample_optimized.cpp.

This sources\gst-plugins\gst-dsexample\gstdsexample_optimized.cpp . source code and follow link is familar for me, could you @yuweiw reproduce it by tgz ? I change osd_sink_pad_buffer_probe() only, make && ./deepstream-test1-app dstest1_config.yml
dp6.3-test1.tar.gz (18.9 KB)

Deepstream sample code snippet - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

As I said before, it’s a format mismatch. Please refer to my
deepstream_test1_app.c (13.7 KB)
file to change the video format to RGBA first.

Thanks for replay, the file is same with I offered(no transfrom). I try to transfrom nv12 to rgba with NvBufSurfTransform api, it work well at deepstream-test1.
But my application the origin(osd sink probe) format already is rgba,I try NvBufSurfTransform change format rgba to rgba, when cv::Mat output_mat = rgba_mat.clone(); , host memory increase fastly, if not clone host memory is stable . My pipeling is appsrc(NV12)->nvvidconv->caps_filter->nvstreammux->nvinfer->nvvidconv->nvdsosd->fakesink, I have no idea about this question, suface info:surfaceList[0], width 1920 height 1080 pitch 7680 layout 0 colorFormat 19 bufferDesc 280 dataSize 8388608 dataPtr 281470815988960. Can you help me?

You can refer to the source code I attached. I added a caps_filter after the nvvideoconvert to change the image format to RGBA. You can try that in your pipeline instead of using the NvBufSurfTransform .

  GstElement *caps_filter = NULL;
  caps_filter = gst_element_factory_make ("capsfilter", "capsfilter");
  GstCaps *caps = NULL;
  GstCapsFeatures *feature = NULL;
  caps =
      gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING,
      "RGBA", NULL);
  feature = gst_caps_features_new ("memory:NVMM", NULL);
  gst_caps_set_features (caps, 0, feature);
  g_object_set (G_OBJECT (caps_filter), "caps", caps, NULL);
  gst_caps_unref (caps);

Further testing, I success reproduce host memeory incease fastly at deepstream-test1 by add code:
g_object_set(G_OBJECT(nvosd), "process-mode", OSD_PROCESS_MODE, "display-text", OSD_DISPLAY_TEXT, NULL);
By default, OSD process-mode is set to GPU_MODE. To change mode, set as:

  • 0: CPU mode
  • 1: GPU mode

when set process-mode to 0 memory incerase very fast and format is RGBA, set to 1 format is NV12 , deepstream-6.0 is all ok, maybe is bug?
I will try add capsfilter as you mentioned, is nvinfer->nvvideoconvert->capsfilter->nvosd ?

Yes. You can refer to the source code I attached.

It works well when process-mode set to 1, thank you.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.