Issue with plotting on image in deepstream-app

I am plotting rectangles on images in deepstream-app using gstdsexample.
gstdsexample is added inside config file as

[ds-example]
enable=1
#processing-width=960
#processing-height=544
full-frame=1
blur-objects=0
unique-id=15

Then I have access to cv::Mat as

cv::Mat in_mat = cv::Mat (surface->surfaceList[frame_meta->batch_id].planeParams.height[0], surface->surfaceList[frame_meta->batch_id].planeParams.width[0], CV_8UC4,
		                  surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0], surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);

Then I can plot on it.

But I have issues.
The image size is 1920x1080. I have four streams and all images from four streams have 1920x1080 image size.
Please see in the attached image.
xmin,ymin,xmax,ymax points don’t match to image size 1920,1080.
First subplot’s xvalues 350 and 410 reach to right side of image.
Second subplot’s xmax-value 360 reaches near the end of image.
But for the fourth image, x values need to be 1370.

Why pixel coordinates don’t match to image size?

The rectangle color is also different.
The color I chose for rectangle was

cv::Scalar( 0, 128, 0, 0.1 )

cv::rectangle( in_mat, cv::Point( dsexample->refboxes[source].corners[boxidx+0], dsexample->refboxes[source].corners[boxidx+1] ),  
                       cv::Point( dsexample->refboxes[source].corners[boxidx+2], dsexample->refboxes[source].corners[boxidx+3]), cv::Scalar( 0, 128, 0, 0.1 ), cv::FILLED);

But instead of plain color, it shows vertical lines.
What is wrong with these issues?

refer DeepStream SDK FAQ - #18 by bcao

I understand the code.
But I don’t understand why boxes’ coordinates don’t match to image size 1920x1080.
Then now I change display color to

  cv::Mat roi = in_mat(cv::Rect(x1, y1, x2-x1, y2-y1 )); 
  cv::Mat color(roi.size(), CV_8UC4, cv::Scalar( 0, 0, 255 ));
  double alpha = 0.3;
  cv::addWeighted(color, alpha, roi, 1.0 - alpha, 0.0, roi);

This code has more control on transparency.
But still color doesn’t display as the color in RGB values.

cv::Mat will do the operation on CPU side, I would suggest you to use the API nvidia provide, refer NvBufSurfTransformCompositeBlend NvDsBufSurfTransform API — Deepstream Deepstream Version: 5.1 documentation

Ic thanks. Let me explore.

Is there a sample how to use this api?
NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend (NvBufSurface * src0 , NvBufSurface * src1 , NvBufSurface * alpha , NvBufSurface * dst , NvBufSurfTransformCompositeBlendParams * blend_params )

It is hard to find params.
I can’t find how to set parameters for this structure NvBufSurfTransformCompositeBlendParams.

We don’t have an example currently.

So how can I use? No example and even no proper documentation. Where can I find parameters for this structure NvBufSurfTransformCompositeBlendParams?

Refer _NvBufSurfTransformCompositeBlendParams — Deepstream Deepstream Version: 5.1 documentation , all the public API/data structure can be found in that link and also you can search the source code, we will consider to add a sample later, but currently you need to refer the resource what already exsist there.

I prefer using nvdosd. Can I use nvdosd from gst-dsexample?
I like to add mask_params to nvdsosd->frame_mask_params->mask_params_list.
If possible, do you have sample?

I would not suggest you to do it in that way, it’s not easy to implement and debug the issue you will meet in that way compared with call the API DS provided

So can you advise me how I can set parameters for this api?
NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend (NvBufSurface * src0 , NvBufSurface * src1 , NvBufSurface * alpha , NvBufSurface * dst , NvBufSurfTransformCompositeBlendParams * blend_params )

My implementation in opencv is

//Get buffer and wrap in cv::Mat

cv::Mat in_mat = cv::Mat (surface->surfaceList[frame_meta->batch_id].height, surface->surfaceList[frame_meta->batch_id].width, CV_8UC4,
		                  surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0], surface->surfaceList[frame_meta->batch_id].pitch);

//Define region of interest to draw mask
cv::Mat roi = in_mat(cv::Rect(x1, y1, x2-x1, y2-y1 ));
//Make a color rectangle, same size as roi
cv::Mat color(roi.size(), CV_8UC4, cv::Scalar( 0, 0, 255 ));
double alpha = 0.3;
//then blend
cv::addWeighted(color, alpha, roi, 1.0 - alpha, 0.0, roi);

For that implementation, how can I set parameters for the api?

Hi May I have reply?

Hi any reply for this? I deployed this application. The memory issue is the one I still need to resolve.

please provide the setup info as other topic does

Thanks this is setup infos.
• Hardware Platform (Jetson / GPU)
Xavier
• DeepStream Version
5.1 (;atest)
• JetPack Version (valid for Jetson only)
R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020
• TensorRT Version
latest
• NVIDIA GPU Driver Version (valid for GPU only)
10.2

I am doing as follows.

NvBufSurface* roi_surface;
NvBufSurface* color_surface;
NvBufSurfaceCreateParams create_params;
create_params.gpuId  = dsexample->gpu_id;  
create_params.size = 0;
create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
create_params.layout = NVBUF_LAYOUT_PITCH;
create_params.memType = NVBUF_MEM_DEFAULT;
create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
create_params.width  = 100;
create_params.height = 200;
cv::Mat roi = in_mat(cv::Rect(x1, y1, x2-x1, y2-y1 )); 
cv::Mat color(roi.size(), CV_8UC4, cv::Scalar( 0, 0, 255 ));
double alpha = 0.3;
if (NvBufSurfaceCreate (&roi_surface, 1, &create_params) != 0) {
   printf ("Error: in Create\n");
}
NvBufSurfaceMemSet (roi_surface, 0, 0, 0);
if (NvBufSurfaceMap (roi_surface, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
	printf("Err in Map\n");
 }
 if (NvBufSurfaceSyncForCpu (roi_surface, 0, 0) !=0) {
	printf("Err in Synccpu");
 }
memcpy(roi_surface->surfaceList[0].mappedAddr.addr[0], roi.ptr(), create_params.width * create_params.height * 4);
if (NvBufSurfaceCreate (&color_surface, 1, &create_params) != 0) {
	 printf ("Error: in Create\n");
 }
NvBufSurfaceMemSet (color_surface, 0, 0, 0);
if (NvBufSurfaceMap (color_surface, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
	printf("Err in Map\n");
 }
 if (NvBufSurfaceSyncForCpu (color_surface, 0, 0) !=0) {
	 printf("Err in Synccpu");
 }
 memcpy(color_surface->surfaceList[0].mappedAddr.addr[0], color.ptr(), create_params.width * create_params.height * 4);
 _NvBufSurfTransformCompositeBlendParams *params;
params->composite_blend_flag=1;
params->input_buf_count=1;
params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], 
                       color_surface->surfaceList[0].mappedAddr.addr[0], alpha, 
                       roi_surface->surfaceList[0].mappedAddr.addr[0], &params);

When compiled, I have errors as

gstdsexample.cpp:1017:32: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
                        params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                ^~~~~~~~
gstdsexample.cpp:1017:57: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
                        params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                                         ^~~~~~~~
gstdsexample.cpp:1017:83: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
          params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                                                     ^~~~~~~~
gstdsexample.cpp:1017:110: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
 top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                                                     ^~~~~~~~
gstdsexample.cpp:1018:102: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
 ransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha,
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1018:152: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
 ].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha,
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1019:112: error: cannot convert ‘double’ to ‘NvBufSurface*’ for argument ‘3’ to ‘NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend(NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurfTransformCompositeBlendParams*)’
                       roi_surface->surfaceList[0].mappedAddr.addr[0], &params);
                                                                              ^
Makefile:74: recipe for target 'gstdsexample.o' failed

Why no member named ‘src_rect’?
How to set parameters for NvBufSurfTransformCompositeBlend?

Thanks for the info and exploring the usage!

According to the usage description of NvBufSurfTransformCompositeBlend(),

the src0, src1, dst and alpha are NvBufSurface.
blend_params can be left un-initialized.

NvBufSurfTransform_ErrorNvBufSurfTransformCompositeBlend(NvBufSurface *src0, NvBufSurface *src1, NvBufSurface *alpha, NvBufSurface *dst, NvBufSurfTransformCompositeBlendParams *blend_params)

Thanks!

I provide NvBufSurface, but I have errors as

gstdsexample.cpp: In function ‘GstFlowReturn gst_dsexample_transform_ip(GstBaseTransform*, GstBuffer*)’:
gstdsexample.cpp:1029:102: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                        NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha_surface->surfaceList[0].mappedAddr.addr[0],
                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1029:152: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                        NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha_surface->surfaceList[0].mappedAddr.addr[0],
                                                                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1029:202: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                       NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha_surface->surfaceList[0].mappedAddr.addr[0],
                                                                                                                                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1030:102: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                                                         roi_surface->surfaceList[0].mappedAddr.addr[0]);
                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1030:103: error: too few arguments to function ‘NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend(NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurfTransformCompositeBlendParams*)’
                                                         roi_surface->surfaceList[0].mappedAddr.addr[0]);

Un-initialization gives me too few parameters error.

why is it still “roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0]”? Shouldn’t be roi_surface ?