The whole application is freezed in saving jpg image in deepstream_parallel_inference_app

[edit_or]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Orin 64G development kit
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only) 6.1
I am trying to save jpg image in deepstream_parallel_inference_app.
Added the following code into osd_sink_pad_buffer_probe.

GstMapInfo in_map_info;
memset (&in_map_info, 0, sizeof (in_map_info));
  if (!gst_buffer_map (buf, &in_map_info, GST_MAP_READ)) {
    g_print ("Error: Failed to map gst buffer\n");
    return GST_PAD_PROBE_OK;
  }
  surface = (NvBufSurface *) in_map_info.data;

Then the whole application is freezed.
osd_sink_pad_buffer_probe probe is attached to tiled_display_bin as

    osd_sink_pad = gst_element_get_static_pad(pipeline->tiled_display_bin.tiler, "sink");
    if (!osd_sink_pad)
      g_print("Unable to get sink pad\n");
    else {
      gst_pad_add_probe(osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
		        osd_sink_pad_buffer_probe, NULL, NULL);
      LatencyCtx *ctx = (LatencyCtx *)g_malloc0(sizeof(LatencyCtx));
      ctx->lock = (GMutex *)g_malloc0(sizeof(GMutex));
      ctx->num_sources = config->num_source_sub_bins;
      gst_pad_add_probe (osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
	  latency_measurement_buf_prob, ctx, NULL);
    }

I like to have bounding boxes in the saved images. Since tiled_display_bin is last stage in the pipeline after osd has plotted bounding boxes, it is considered the bounding boxes are in the saved images.
That is the reason, jpg is saved in the osd_sink_pad_buffer_probe.

The whole loop is shown below.

static GstPadProbeReturn
osd_sink_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                          gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *)info->data;
  guint num_rects = 0;
  NvDsObjectMeta *obj_meta = NULL;
  NvDsMetaList *l_frame = NULL;
  NvBufSurface *surface = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsDisplayMeta *display_meta = NULL;
  gboolean is_first_object = TRUE;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);
  GstMapInfo in_map_info;
  std::vector<int> cams;  
  zones_display.get_cameras(cams);
  memset (&in_map_info, 0, sizeof (in_map_info));
  if (!gst_buffer_map (buf, &in_map_info, GST_MAP_READ)) {
    g_print ("Error: Failed to map gst buffer\n");
    return GST_PAD_PROBE_OK;
  }
  surface = (NvBufSurface *) in_map_info.data;
  int offset = 0;
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    is_first_object = TRUE;
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);    
    bool saved=false;
    for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next)
    {
        obj_meta = (NvDsObjectMeta *)(l_obj->data);
        //cout << "source id " << frame_meta->source_id << " id " << obj_meta->object_id << " class " << obj_meta->class_id << endl;
        if(saved==false){
            NvDsObjEncUsrArgs frameData = { 0 };
	    frameData.isFrame = 1;
	    frameData.saveImg = true;
	    frameData.attachUsrMeta = false;
	    frameData.scaleImg = FALSE;
            std::string str = "source_"+std::to_string(frame_meta->source_id)+".jpg"; 
            strcpy(frameData.fileNameImg, str.c_str()); 
	    frameData.scaledWidth = 0;
	    frameData.scaledHeight = 0;
	    frameData.quality = 80;
	    saved=true;
	    nvds_obj_enc_process (obj_ctx_handle, &frameData, surface, NULL, frame_meta);
	}
    }
  }
  nvds_obj_enc_finish (obj_ctx_handle);
  //frame_number++;
  return GST_PAD_PROBE_OK;
}

[yuweiw]

@edit_or

Since tiled_display_bin is last stage in the pipeline after osd has plotted bounding boxes, it is considered the bounding boxes are in the saved images.

No. You can refer to our pipeline introduction, the nvdsosd plugin is behind the tiler plugin.

[edit_or]

Then why it is freezed?

[yuweiw]

Can you narrow it down by following these steps?

1. You can add some log to find which line of code is causing the problem.
2. You can try to add your code to the probe of the nvdsosd plugin or pgie plugin and try that.
3. You can add your code to our deepstream_test3_app.c to verify if it’s related to the parallel sample.

[edit_or]

Yes you are right. It is not freezed in nvdsosd plugin.
But the issue in nvdsosd plugin is that combined image is saved instead of individual image for four cctv sources. The saved image is shown below.

Inside the following loop in osd_sink_pad_buffer_probe, I have four sources. So it should run four time. But it runs through only one time. That means, I can have meta infos for the first source only. How can I split for four different sources.

for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {

}

[yuweiw]

This is normal because there are no batches after the nvmultistreamtiler plugin anymore. So if you want to save 4 images, you need to add that probe function before the nvmultistreamtiler plugin.

[edit_or]

It is in tiler plugin? But the application freezed in saving images
in the probe attached to tiler plugin.

[edit_or]

The following lines of codes freezed the application.

if (!gst_buffer_map (buf, &in_map_info, GST_MAP_READ)) {
    g_print ("Error: Failed to map gst buffer\n");
    return GST_PAD_PROBE_OK;
  }

tiler_sink_pad_buffer_probe is attached to tile_display_bin.tiler.

tiler_sink_pad = gst_element_get_static_pad(pipeline->tiled_display_bin.tiler, "sink");
    if (!tiler_sink_pad)
      g_print("Unable to get sink pad\n");
    else {
      gst_pad_add_probe(tiler_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
		        tiler_sink_pad_buffer_probe, NULL, NULL);
      LatencyCtx *ctx = (LatencyCtx *)g_malloc0(sizeof(LatencyCtx));
      ctx->lock = (GMutex *)g_malloc0(sizeof(GMutex));
      ctx->num_sources = config->num_source_sub_bins;
      gst_pad_add_probe (tiler_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
	  latency_measurement_buf_prob, ctx, NULL);
    }

All fps are freezed like that

|**PERF: 0.00 (0.00)|0.00 (0.00)|0.00 (0.00)|0.00 (0.00)||
|---|---|---|---|---|
|**PERF: 53.65 (6.23)|53.65 (6.23)|53.65 (6.23)|53.65 (6.23)||
|**PERF: 0.00 (2.02)|0.00 (2.02)|0.00 (2.02)|0.00 (2.02)||
|**PERF: 0.00 (1.21)|0.00 (1.21)|0.00 (1.21)|0.00 (1.21)||
|**PERF: 0.00 (0.86)|0.00 (0.86)|0.00 (0.86)|0.00 (0.86)||
|**PERF: 0.00 (0.67)|0.00 (0.67)|0.00 (0.67)|0.00 (0.67)||
|**PERF: 0.00 (0.55)|0.00 (0.55)|0.00 (0.55)|0.00 (0.55)||
|**PERF: 0.00 (0.46)|0.00 (0.46)|0.00 (0.46)|0.00 (0.46)||

[edit_or]

Any suggestion for this?

[yuweiw]

Could you try to modify your code like below? You need to unmap the in_map_info after using that.

  surface = (NvBufSurface *) in_map_info.data;
  gst_buffer_unmap (buf, &in_map_info);