Add custom text label for rendering with NVOsd

Hello
I am running a pipeline based on Deepstream 3.0 samples

I have added custom lines on osd_sink_pad_buffer_probe using :

NvDsMeta *gst_line_meta_in = NULL;
NvDsLineMeta *line_meta_in = (NvDsLineMeta *) g_malloc0 (sizeof (NvDsLineMeta));

line_meta_in->batch_id = frame_meta->batch_id;
generate_line_ingresso(line_meta_in,1.0,0,0,1.0,X1,Y1,X2,Y2,0.0,0,1.0,1.0,X3,Y3,X4,Y4);

gst_line_meta_in = gst_buffer_add_nvds_meta (buf, (void*)line_meta_in,(GDestroyNotify)nvds_free_line_info);

if (gst_line_meta_in) 
{
     gst_line_meta_in->meta_type = NVDS_META_LINE_INFO;
} else {
     g_print ("Error in attaching event meta to buffer\n");
}

How can I add a custom label text (not refferred to NvDsObjectParams) and render with Nvosd same way as for lines ?

Any suggestion welcome
Thanx in advance
Marco

Set appCtx[0]->show_bbox_text = TRUE; in deepstream_app_main.c before g_main_loop_new

Hi Chris,
Thanks for your fast reply.

I have added line in deepstream_app_main.c before g_main_loop nvosd now shows labels on detected objects.
My goal is to show always a simple txtlabel on screen not reffered to detected objects.
Just a simple static text label on top of screen to show average speed of object.
In example before I was able to add static lines and render them on osd display adding the code on on osd_sink_pad_buffer_probe.
Is there a way to do similar thing to render a static label text on top of screen.

Thnxxxx
Marco

Hello all,
finally I figured out how to add a custom label box on osd display not related to detected objects.
I think this is not a good way to achive this, maybe more correct and elegant ways are possible.
I created on osd_sink_pad_buffer_probe a fake NvDsMeta filled with all necassery info with a fake object inside containg the label I wish to show and add it to buffer.
Here it’s the code maybe could be useful for someone :

  1. Delcared a static custom metadata cleaning function
/**
	 * Free the custom meta data allocated
	 */
	 static void free_ds_meta (gpointer meta_data)
	 {
		 NvDsFrameMeta* params = (NvDsFrameMeta *) meta_data;
		 for (guint i = 0; i < params->num_rects; i++) {
			 g_free (params->obj_params[i].text_params.display_text);
		 }

		 g_free (params->obj_params);
		 g_free (params);
	 }
  1. On osd_sink_pad_buffer_probe added the following code :
NvDsMeta *fake_dsmeta;
NvDsFrameMeta *fake_bbparams = (NvDsFrameMeta *) g_malloc0 (sizeof (NvDsFrameMeta));
fake_bbparams->obj_params = (NvDsObjectParams *) g_malloc0 (sizeof (NvDsObjectParams));
fake_bbparams->gie_type = 3;
fake_bbparams->nvosd_mode = NV_OSD_MODE_GPU;
fake_bbparams->batch_id = frame_meta->batch_id;
NvDsObjectParams *obj_param = &fake_bbparams->obj_params[0];
NvOSD_RectParams &rect_params = obj_param->rect_params;
NvOSD_TextParams &text_params = obj_param->text_params;
rect_params.left = 0;
rect_params.top = 0;
rect_params.width = 0;
rect_params.height = 0;
rect_params.has_bg_color = 0;
rect_params.bg_color = (NvOSD_ColorParams) {1, 1, 0, 0.4};
rect_params.border_width = 0;
rect_params.border_color = (NvOSD_ColorParams) {1, 0, 0, 1};
fake_bbparams->num_rects++;
obj_param->has_new_info = TRUE;
obj_param->class_id = 0;
obj_param->tracking_id = -1;
obj_param->attr_info[0].is_attr_label = 1;
string text_out = "PROVA LABEL !!!!!";
text_params.display_text = g_strdup (text_out.c_str());
text_params.x_offset = 100;
text_params.y_offset = 100;
text_params.set_bg_clr = 1;
text_params.text_bg_clr = (NvOSD_ColorParams) {0, 0, 0, 1};
text_params.font_params.font_name = "Arial";
text_params.font_params.font_size = 20;
text_params.font_params.font_color = (NvOSD_ColorParams) {1, 1, 1, 1};
fake_bbparams->num_strings++;

fake_dsmeta = gst_buffer_add_nvds_meta (buf, fake_bbparams, free_ds_meta);
if (fake_dsmeta)
{
   fake_dsmeta->meta_type = NVDS_META_FRAME_INFO;
}
else
{
   g_print ("Error in attaching event meta to buffer\n");
   ERROR << "ATTACCHING CUSTOM FRAME META";
}

Again this is an awful way to achive this and hope someone can help to get a better solution

Regards
Marco Gonnelli

your method is too heavy. we do not have to rely on NvOsd for display.

Extract raw frame data from GstBuffer(NvBufSurface type) and copy to host memeory, then convert to cv::Mat, then draw what you want on cv::Mat using opencv lib, copy back the cv::Mat memory to device memory at last. Maybe this is just what NvOsd plugin do, i do not have NvOsd source code ,just my personal thoughts.

the gst-dsexample plugin show you how to convert the raw frame data to cv::Mat.

Thnxx for reply zhouzhi9

Yes we are planning to do the way u mentioned.
Creating a new plugin based on gst-dsexample to exctract frame from GstBuffer and work on it with OpenCV.
As far from now following gst-dsexample plugin we are able to get Buffer from GPU/GstBuffer copy on host CPU Buffer and create and modify it with OpenCV.
Unfortunately gst-dsexample only works with metadata modification and we are stuck on getting back modified cv::Mat cpu buffer on device Gpu Memory and NvBufSurface pointer in output GstBuffer.

Would be great to have NvOsdPlugin source code… :-(

try to cudamemcpy dsexample->host_rgb_buf back to surface->buf_data[i] (gst-dsexample demo in DeepStream 3.0 Release), not validated, please let me know if you succeed.

In principle, we can modify meta data and ALSO frame data in our custom plugin or probe function.

Hello zhouzhi9,

We have implemented code following ur suggestion, both on custum plugin and probe function and it works goood :-)

On custom plugin frame is copied from GPU to CPU using CudaMemCpy, modified with openCV and then copied back on GPU using CudaMemCpy again, frame is modified and other plugin on pipeline can grab modified frame from input buffer.

Here is the code we have used calling this static function on src_osd_buffer_probe in this case :

static void modify_frame(GstBuffer *inGPUbuf,
			 	 	 	 	  int imageW,
			 	 	 	 	  int imageH,
			 	 	 	 	  int batchIndex,
			 	 	 	 	  cudaStream_t npp_stream)
	 {
		 NvBufSurface *surface = NULL;
		 GstMapInfo in_map_info;
		 memset (&in_map_info, 0, sizeof (in_map_info));
		 void *host_rgb_buf;

		 if (!gst_buffer_map (inGPUbuf, &in_map_info, GST_MAP_READ))
		 {
		    ERROR << "MainPipelineThread::modify_frame Failed to map gst buffer";
		 	return;
		 }

		 surface = (NvBufSurface *) in_map_info.data;
		 void *buf_surface = surface->buf_data[batchIndex];

		 CHECK_CUDA_STATUS (cudaMallocHost (&host_rgb_buf,
		 				imageW * imageH * RGBA_BYTES_PER_PIXEL), "Could not allocate cuda host buffer");

		 CHECK_CUDA_STATUS (cudaMemcpy(host_rgb_buf,buf_surface,
		 							 imageW * imageH * RGBA_BYTES_PER_PIXEL,cudaMemcpyDeviceToHost),
		 					 	     "Error copy Device to Host");

		 cv::Mat cvmat(cv::Size(imageW,imageH),CV_8UC4,host_rgb_buf,imageW * RGBA_BYTES_PER_PIXEL);

		 //DO YOUR OPENCV ELABORATION ON FRAME HERE
		 cv::rectangle(cvmat,cv::Rect(10, 10,20,20),cv::Scalar(0,0,255), 3);
		 ///

		 CHECK_CUDA_STATUS (cudaMemcpy(
				 buf_surface,
				 cvmat.data,
				 cvmat.cols * cvmat.rows * RGBA_BYTES_PER_PIXEL,cudaMemcpyHostToDevice),
		 	 "Error copy Host to Device");

		 CHECK_CUDA_STATUS (cudaStreamSynchronize (npp_stream), "Failed to synchronize cuda stream");


	 }

happy to hear that!

Hello,

I am trying to change something in frame_meta and add the modified frame_meta to buffer. However, I end up in failure and the frame_meta show in next plugin remain same. Is there any mistake in my code?

I wish that the change I did in this plugin can reused again in the next plugin. But, now, all the frame meta will remain unchanged when I add a probe in the tiler.

NvDsMeta *gst_frame_meta = NULL;
    NvDsFrameMeta *frame_meta;
    frame_meta = (NvDsFrameMeta *)dsmeta->meta_data;
    frame_meta->num_rects = 1;
    NvOSD_RectParams *rect_params = (NvOSD_RectParams *)&frame_meta->obj_params[0].rect_params;
    rect_params.left = 100;
    
    gst_frame_meta = gst_buffer_add_nvds_meta(outbuf, frame_meta, free_frame_meta);
    if (gst_frame_meta)
    {
      gst_frame_meta->meta_type = NVDS_META_FRAME_INFO;
    }
    else
    {
      g_print("Error in attaching frame meta to buffer\n");
    }
static void free_frame_meta(gpointer meta_data)
{
  NvDsFrameMeta *params = (NvDsFrameMeta *)meta_data;
  for (guint i = 0; i < params->num_rects; i++)
  {
    g_free(params->obj_params[i].text_params.display_text);
  }
  g_free(params->obj_params);
  g_free(params);
}