How to save frame as jpg and send the filename to kafka in deepstream 5.0 deepstream_test5

• Hardware Platform - Jetson TX2
• DeepStream Version - 5.0

I am trying to refer opencv_test for saving image,

In deepstream_test5 (for multiple camera - tiled enabled) , it blurs out the objects in all cameras, but when i left-click on the particular camera to view, i get blank display,

And when i enable DSEXAMPLE_DEBUG to save image using cv::imwrite inside gstdsexample.cpp, i get segmentation fault error while running deepstream_test5, i get error on cv::imwrite

Without converting to RGBA , dsexample works perfectly (without blur objects), is this issue about converting to RGBA?

Below are the changes that i did so far,
deepstream_test5_config.txt

[ds-example]
enable=1
processing-width=640
processing-height=480
full-frame=0
blur-objects=1
gpu-id=0
nvbuf-memory-type=0

I have added blur-objects in deepstream_config_file_parser.c, and then i have set it in deepstream_dsexample.c

In deepstream_app.c,

if (config->dsexample_config.enable) {
    // Create dsexample element bin and set properties
    if (!create_dsexample_bin (&config->dsexample_config,
            &pipeline->dsexample_bin)) {
      goto done;
    }
      /* Set properties of the caps_filter element */
      caps_filter = gst_element_factory_make ("capsfilter", NULL);
  GstCaps *caps =
      gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING, "RGBA",
      NULL);
  GstCapsFeatures *feature = gst_caps_features_new (MEMORY_FEATURES, NULL);
  gst_caps_set_features (caps, 0, feature);

  g_object_set (G_OBJECT (caps_filter), "caps", caps, NULL);
  gst_bin_add (GST_BIN (pipeline->pipeline), caps_filter);
    NVGSTDS_LINK_ELEMENT (caps_filter, last_elem);
    last_elem = caps_filter;
    // Add dsexample bin to instance bin
    gst_bin_add (GST_BIN (pipeline->pipeline), pipeline->dsexample_bin.bin);

    // Link this bin to the last element in the bin
    NVGSTDS_LINK_ELEMENT (pipeline->dsexample_bin.bin, last_elem);

    // Set this bin as the last element
    last_elem = pipeline->dsexample_bin.bin;
  }

Hi,
You can customize the metadata to put the JPEGs. For saving to JPEGs, please refer to

Tried second link (to save full frame), Still the same error , segmentation fault occurs when i use cv::imwrite()

Hi,
You would need to send frames in BGR format in calling cv::imwrite().

cv::cvtColor (in_mat, out_mat, CV_RGBA2BGR);
cv::imwrite(filename, out_mat);

Yes i am doing that,
cv::cvtColor (in_mat, bgr_frame, cv::COLOR_RGBA2BGR);
cv::imwrite(filename,bgr_frame);

Could you tell me on how to convert frame to RGBA in deepstream_app.c for multiple streams??

Hi,
The sample code is get_converted_mat() in

deepstream-5.0\sources\gst-plugins\gst-dsexample\gstdsexample.cpp

It convert NV12(ip_surf) to RGBA(dsexample->inter_buf) in

err = NvBufSurfTransform (&ip_surf, dsexample->inter_buf, &transform_params);

NvBufSurfTransform is not converting frame to RGBA

/* Transformation scaling+format conversion if any. */
  err = NvBufSurfTransform (&ip_surf, dsexample->inter_buf, &transform_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    goto error;
  }
  else{ g_print("\nxxxxxxx %d %d\n",dsexample->video_info.finfo->format,GST_VIDEO_FORMAT_RGBA);
}

Output:
xxxxxxx 23 11

I am still getting segmentation fault error on cv::imwrite

Could you please share any sample code which can save image frames for multiple sources in deepstream_app.c ??

Hi,
You probably don’t create dsexample->inter_buf in NVBUF_COLOR_FORMAT_RGBA. Please refer to the code of creating the buffer:

  /* An intermediate buffer for NV12/RGBA to BGR conversion  will be
   * required. Can be skipped if custom algorithm can work directly on NV12/RGBA. */
  create_params.gpuId  = dsexample->gpu_id;
  create_params.width  = dsexample->processing_width;
  create_params.height = dsexample->processing_height;
  create_params.size = 0;
  create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
  create_params.layout = NVBUF_LAYOUT_PITCH;
#ifdef __aarch64__
  create_params.memType = NVBUF_MEM_DEFAULT;
#else
  create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
#endif

  if (NvBufSurfaceCreate (&dsexample->inter_buf, 1,
          &create_params) != 0) {
    GST_ERROR ("Error: Could not allocate internal buffer for dsexample");
    goto error;
  }

I am doint that also , inside gstbasetransform_class->start = GST_DEBUG_FUNCPTR (gst_dsexample_start);

Below code inside deepstream_app.c is converting to rgba

 /* Set properties of the caps_filter element */
            caps_filter = gst_element_factory_make ("capsfilter", NULL);
        GstCaps *caps = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING, "RGBA",NULL);
        GstCapsFeatures *feature = gst_caps_features_new (MEMORY_FEATURES, NULL);
        gst_caps_set_features (caps, 0, feature);

        g_object_set (G_OBJECT (caps_filter), "caps", caps, NULL);
        gst_bin_add (GST_BIN (pipeline->pipeline), caps_filter);
          NVGSTDS_LINK_ELEMENT (caps_filter, last_elem);
          last_elem = caps_filter;
  gst_caps_unref (caps);

But getting segmentation fault while saving image

If i comment out the cv::imwrite() then it blur blurs the objects but tiler not working as expected, if i left click on display to open particular camera, it shows blank

Hi,
You may check if you can apply

to gie_processing_done_buf_prob() in deepstream_app.c. We have verified it and you should have it work first.

1 Like

how do i use opencv library in deepstream_app.c?

Hi,
You need to include necessary header files and modify Makefile

Header files:

/* Open CV headers */
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"

#include "nvbufsurface.h"
#include "nvbufsurftransform.h"

Makefile:

I am getting below error!!

make
cc -c -o deepstream_test5_app_main.o -DPLATFORM_TEGRA -I…/…/apps-common/includes -I /usr/local/cuda-10.2/include -I…/…/…/includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=5 pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11 opencv4 deepstream_test5_app_main.c
In file included from /usr/include/opencv4/opencv2/imgproc.hpp:46:0,
from /usr/include/opencv4/opencv2/imgproc/imgproc.hpp:48,
from deepstream_app.h:49,
from deepstream_test5_app_main.c:32:
/usr/include/opencv4/opencv2/core.hpp:49:4: error: #error core.hpp header must be compiled as C++
# error core.hpp header must be compiled as C++
^~~~~
In file included from /usr/include/opencv4/opencv2/core.hpp:52:0,
from /usr/include/opencv4/opencv2/imgproc.hpp:46,
from /usr/include/opencv4/opencv2/imgproc/imgproc.hpp:48,
from deepstream_app.h:49,
from deepstream_test5_app_main.c:32:
/usr/include/opencv4/opencv2/core/cvdef.h:690:4: error: #error “OpenCV 4.x+ requires enabled C++11 support”
# error “OpenCV 4.x+ requires enabled C++11 support”
^~~~~
/usr/include/opencv4/opencv2/core/cvdef.h:695:10: fatal error: array: No such file or directory
#include
^~~~~~~
compilation terminated.
Makefile:58: recipe for target ‘deepstream_test5_app_main.o’ failed
make: *** [deepstream_test5_app_main.o] Error 1

Hi,
You would need to refer to the Makefile and try to fix the compiling error.

An easier way is to enable dsexample in config file. You have mentioned you hit segment fault. Does it happen in running default sample? We have verified the default sample and this should not happen. You would need to check this first. To run the default dsexample successfully.

No this doesnt happen when running default app and even if dsexample is enabled

It happens only when i try to save images cv::imwrite

In gstdsexample.cpp , saving images is disabled, i.e. DSEXAMPLE_DEBUG is not defined.
Saving images works well with opencv-test-app but not with deepstream-test5

////#ifdef DSEXAMPLE_DEBUG
        /* Use openCV to remove padding and convert RGBA to BGR. Can be skipped if
        * algorithm can handle padded RGBA data. */
#if (CV_MAJOR_VERSION >= 4)
        cv::cvtColor (in_mat, *dsexample->cvmat, cv::COLOR_RGBA2BGR);
#else
        cv::cvtColor (in_mat, *dsexample->cvmat, CV_RGBA2BGR);
#endif
        /* used to dump the converted mat to files for debug */
        static guint cnt = 0;
        cv::imwrite("out_" + std::to_string (cnt) + ".jpeg", *dsexample->cvmat);
        cnt++;
////#endif

Hi,
We tried the following code and it works well. Pleas give it a try.

  static gint dump = 0;
  if (dump < 150) {
      char filename[64];
      snprintf(filename, 64, "/tmp/image%03d.jpg", dump);
      cv::imwrite(filename, *dsexample->cvmat);
      dump++;
  }

Hi,
Please also try deepstream-app. If you still hit segment fault in running deepstrem-app, please consider to do clean re-flash through SDKManager. We have verified the working well. A bit strange that the same patch does not work on your system. May help to do clean re-flash.

I am able to save the images, now i would like to send filename to kafka

how do i attach filename to metadata from gstdsexample.cpp?

I referred deepstream_user_metadata_app.c

Not able to retrieve the data from deepstream_test5_app_main.c

Hi,
You would need to customize the metadata, please refer to

In gstdsexample.cpp, i have added below code

static gpointer meta_copy_func (gpointer data, gpointer user_data)
{
  NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
  NvDsEventMsgMeta *srcMeta = (NvDsEventMsgMeta *) user_meta->user_meta_data;
  NvDsEventMsgMeta *dstMeta = (NvDsEventMsgMeta *) g_malloc (sizeof (NvDsEventMsgMeta));

  //dstMeta = g_memdup (srcMeta, sizeof (NvDsEventMsgMeta));

  if (srcMeta->frame_image){
    dstMeta->frame_image = g_strdup (srcMeta->frame_image);  }

  return dstMeta;
}


static void meta_free_func (gpointer data, gpointer user_data)
{
  NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
  NvDsEventMsgMeta *srcMeta = (NvDsEventMsgMeta *) user_meta->user_meta_data;
  user_meta->user_meta_data = NULL;

  if (srcMeta->frame_image) {
    g_free (srcMeta->frame_image);
  }

  delete srcMeta;
}



static GstFlowReturn
gst_dsexample_transform_ip (GstBaseTransform * btrans, GstBuffer * inbuf)
{
  GstDsExample *dsexample = GST_DSEXAMPLE (btrans);
  GstMapInfo in_map_info;
  GstFlowReturn flow_ret = GST_FLOW_ERROR;
  gdouble scale_ratio = 1.0;
  // DsExampleOutput *output;

  NvBufSurface *surface = NULL;
  NvDsBatchMeta *batch_meta = NULL;
  NvDsFrameMeta *frame_meta = NULL;
  NvDsMetaList * l_frame = NULL;
  guint i = 0;

  NvDsUserMeta *user_meta = NULL;   
  NvDsMetaType user_meta_type = NVDS_USER_FRAME_META_EXAMPLE;  

  dsexample->frame_num++;
  CHECK_CUDA_STATUS (cudaSetDevice (dsexample->gpu_id),
      "Unable to set cuda device");

  memset (&in_map_info, 0, sizeof (in_map_info));
  if (!gst_buffer_map (inbuf, &in_map_info, GST_MAP_READ)) {
    g_print ("Error: Failed to map gst buffer\n");
    goto error;
  }

  surface = (NvBufSurface *) in_map_info.data;
  GST_DEBUG_OBJECT (dsexample,
      "Processing Frame %" G_GUINT64_FORMAT " Surface %p\n",
      dsexample->frame_num, surface);

  if (CHECK_NVDS_MEMORY_AND_GPUID (dsexample, surface))
    goto error;

  batch_meta = gst_buffer_get_nvds_batch_meta (inbuf);
  if (batch_meta == nullptr) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvDsBatchMeta not found for input buffer."), (NULL));
    return GST_FLOW_ERROR;
  }

  if (dsexample->process_full_frame) {
        NvDsMetaList * l_obj = NULL;
    NvDsObjectMeta *obj_meta = NULL;
    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next)
    {
      frame_meta = (NvDsFrameMeta *) (l_frame->data);
      NvOSD_RectParams rect_params;

      /* Scale the entire frame to processing resolution */
      rect_params.left = 0;
      rect_params.top = 0;
      rect_params.width = dsexample->video_info.width;
      rect_params.height = dsexample->video_info.height;

      /* Scale and convert the frame */
      if (get_converted_mat (dsexample, surface, i, &rect_params,
            scale_ratio, dsexample->video_info.width,
            dsexample->video_info.height, frame_meta, batch_meta, user_meta, user_meta_type) != GST_FLOW_OK) {
        goto error;
      }
       for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
          l_obj = l_obj->next)
      {
        
        obj_meta = (NvDsObjectMeta *) (l_obj->data);
        /** Generate NvDsEventMsgMeta for every object */

        NvDsEventMsgMeta *msg_meta =
            (NvDsEventMsgMeta *) g_malloc0 (sizeof (NvDsEventMsgMeta));
        
        char *fileNameString = new char [128];
        
        snprintf (fileNameString, 128, "d.jpg");
        msg_meta->frame_image = (gchar *) g_malloc0 (128);
        strncpy (msg_meta->frame_image, fileNameString, 128);
      
        NvDsUserMeta *user_event_meta =
            nvds_acquire_user_meta_from_pool (batch_meta);
        if (user_event_meta) {
          /*
           * Since generated event metadata has custom objects for
           * Vehicle / Person which are allocated dynamically, we are
           * setting copy and free function to handle those fields when
           * metadata copy happens between two components.
           */
          user_event_meta->user_meta_data = (void *) msg_meta;
          user_event_meta->base_meta.batch_meta = batch_meta;
          user_event_meta->base_meta.meta_type = NVDS_EVENT_MSG_META;
          user_event_meta->base_meta.copy_func = (NvDsMetaCopyFunc) meta_copy_func;
          user_event_meta->base_meta.release_func = (NvDsMetaReleaseFunc) meta_free_func;
          nvds_add_user_meta_to_frame (frame_meta, user_event_meta);
        } else {
          g_print ("Error in attaching event meta to buffer\n");
        }
      }
    
    }

  }
flow_ret = GST_FLOW_OK;

error:
  gst_buffer_unmap (inbuf, &in_map_info);
  return flow_ret;
}

In nvdsmeta_schema.h,

typedef struct NvDsEventMsgMeta {

  gchar *frame_image;
.......
.......
.......
}

I am not able to fetch frame_image from nvmsgconv.cpp

 if (events[0].metadata->frame_image){
    json_object_set_string_member (rootObj, "frame_image", events[0].metadata->frame_image);
    cout << "\n frame_image --> "<< events[0].metadata->frame_image ;
    
  }
  else{
    cout << "\n#frame_image \n " ;
    json_object_set_string_member (rootObj, "frame_image", "");}