Issue with plotting on image in deepstream-app

edit_or · August 8, 2021, 7:43am

I am plotting rectangles on images in deepstream-app using gstdsexample.
gstdsexample is added inside config file as

[ds-example]
enable=1
#processing-width=960
#processing-height=544
full-frame=1
blur-objects=0
unique-id=15

Then I have access to cv::Mat as

cv::Mat in_mat = cv::Mat (surface->surfaceList[frame_meta->batch_id].planeParams.height[0], surface->surfaceList[frame_meta->batch_id].planeParams.width[0], CV_8UC4,
		                  surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0], surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);

Then I can plot on it.

But I have issues.
The image size is 1920x1080. I have four streams and all images from four streams have 1920x1080 image size.
Please see in the attached image.
xmin,ymin,xmax,ymax points don’t match to image size 1920,1080.
First subplot’s xvalues 350 and 410 reach to right side of image.
Second subplot’s xmax-value 360 reaches near the end of image.
But for the fourth image, x values need to be 1370.

Why pixel coordinates don’t match to image size?

The rectangle color is also different.
The color I chose for rectangle was

cv::Scalar( 0, 128, 0, 0.1 )

cv::rectangle( in_mat, cv::Point( dsexample->refboxes[source].corners[boxidx+0], dsexample->refboxes[source].corners[boxidx+1] ),  
                       cv::Point( dsexample->refboxes[source].corners[boxidx+2], dsexample->refboxes[source].corners[boxidx+3]), cv::Scalar( 0, 128, 0, 0.1 ), cv::FILLED);

But instead of plain color, it shows vertical lines.
What is wrong with these issues?

bcao · August 9, 2021, 8:13am

refer DeepStream SDK FAQ - #18 by bcao

edit_or · August 11, 2021, 1:58am

I understand the code.
But I don’t understand why boxes’ coordinates don’t match to image size 1920x1080.
Then now I change display color to

  cv::Mat roi = in_mat(cv::Rect(x1, y1, x2-x1, y2-y1 )); 
  cv::Mat color(roi.size(), CV_8UC4, cv::Scalar( 0, 0, 255 ));
  double alpha = 0.3;
  cv::addWeighted(color, alpha, roi, 1.0 - alpha, 0.0, roi);

This code has more control on transparency.
But still color doesn’t display as the color in RGB values.

bcao · August 19, 2021, 12:47am

cv::Mat will do the operation on CPU side, I would suggest you to use the API nvidia provide, refer NvBufSurfTransformCompositeBlend NVIDIA DeepStream SDK API Reference: Main Page

edit_or · August 19, 2021, 1:16am

Ic thanks. Let me explore.

edit_or · August 19, 2021, 1:41am

Is there a sample how to use this api?
NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend (NvBufSurface * src0 , NvBufSurface * src1 , NvBufSurface * alpha , NvBufSurface * dst , NvBufSurfTransformCompositeBlendParams * blend_params )

It is hard to find params.
I can’t find how to set parameters for this structure NvBufSurfTransformCompositeBlendParams.

bcao · August 23, 2021, 2:19am

We don’t have an example currently.

edit_or · August 23, 2021, 4:00am

So how can I use? No example and even no proper documentation. Where can I find parameters for this structure NvBufSurfTransformCompositeBlendParams?

bcao · August 23, 2021, 6:37am

Refer https://docs.nvidia.com/metropolis/deepstream/sdk-api/BufSurfTransform/_NvBufSurfTransformCompositeBlendParams.html?highlight=nvbufsurftransformcompositeblendparams#_CPPv439_NvBufSurfTransformCompositeBlendParams , all the public API/data structure can be found in that link and also you can search the source code, we will consider to add a sample later, but currently you need to refer the resource what already exsist there.

edit_or · August 23, 2021, 9:39am

I prefer using nvdosd. Can I use nvdosd from gst-dsexample?
I like to add mask_params to nvdsosd->frame_mask_params->mask_params_list.
If possible, do you have sample?

bcao · August 24, 2021, 9:57am

I would not suggest you to do it in that way, it’s not easy to implement and debug the issue you will meet in that way compared with call the API DS provided

edit_or · August 24, 2021, 10:18am

So can you advise me how I can set parameters for this api?
NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend (NvBufSurface * src0 , NvBufSurface * src1 , NvBufSurface * alpha , NvBufSurface * dst , NvBufSurfTransformCompositeBlendParams * blend_params )

My implementation in opencv is

//Get buffer and wrap in cv::Mat

cv::Mat in_mat = cv::Mat (surface->surfaceList[frame_meta->batch_id].height, surface->surfaceList[frame_meta->batch_id].width, CV_8UC4,
		                  surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0], surface->surfaceList[frame_meta->batch_id].pitch);

//Define region of interest to draw mask
cv::Mat roi = in_mat(cv::Rect(x1, y1, x2-x1, y2-y1 ));
//Make a color rectangle, same size as roi
cv::Mat color(roi.size(), CV_8UC4, cv::Scalar( 0, 0, 255 ));
double alpha = 0.3;
//then blend
cv::addWeighted(color, alpha, roi, 1.0 - alpha, 0.0, roi);

For that implementation, how can I set parameters for the api?

edit_or · August 26, 2021, 12:13pm

Hi May I have reply?

edit_or · August 28, 2021, 10:47am

Hi any reply for this? I deployed this application. The memory issue is the one I still need to resolve.

mchi · August 28, 2021, 3:21pm

please provide the setup info as other topic does

edit_or · August 29, 2021, 3:56am

Thanks this is setup infos.
• Hardware Platform (Jetson / GPU)
Xavier
• DeepStream Version
5.1 (;atest)
• JetPack Version (valid for Jetson only)
R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020
• TensorRT Version
latest
• NVIDIA GPU Driver Version (valid for GPU only)
10.2

edit_or · August 31, 2021, 5:33am

I am doing as follows.

NvBufSurface* roi_surface;
NvBufSurface* color_surface;
NvBufSurfaceCreateParams create_params;
create_params.gpuId  = dsexample->gpu_id;  
create_params.size = 0;
create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
create_params.layout = NVBUF_LAYOUT_PITCH;
create_params.memType = NVBUF_MEM_DEFAULT;
create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
create_params.width  = 100;
create_params.height = 200;
cv::Mat roi = in_mat(cv::Rect(x1, y1, x2-x1, y2-y1 )); 
cv::Mat color(roi.size(), CV_8UC4, cv::Scalar( 0, 0, 255 ));
double alpha = 0.3;
if (NvBufSurfaceCreate (&roi_surface, 1, &create_params) != 0) {
   printf ("Error: in Create\n");
}
NvBufSurfaceMemSet (roi_surface, 0, 0, 0);
if (NvBufSurfaceMap (roi_surface, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
	printf("Err in Map\n");
 }
 if (NvBufSurfaceSyncForCpu (roi_surface, 0, 0) !=0) {
	printf("Err in Synccpu");
 }
memcpy(roi_surface->surfaceList[0].mappedAddr.addr[0], roi.ptr(), create_params.width * create_params.height * 4);
if (NvBufSurfaceCreate (&color_surface, 1, &create_params) != 0) {
	 printf ("Error: in Create\n");
 }
NvBufSurfaceMemSet (color_surface, 0, 0, 0);
if (NvBufSurfaceMap (color_surface, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
	printf("Err in Map\n");
 }
 if (NvBufSurfaceSyncForCpu (color_surface, 0, 0) !=0) {
	 printf("Err in Synccpu");
 }
 memcpy(color_surface->surfaceList[0].mappedAddr.addr[0], color.ptr(), create_params.width * create_params.height * 4);
 _NvBufSurfTransformCompositeBlendParams *params;
params->composite_blend_flag=1;
params->input_buf_count=1;
params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], 
                       color_surface->surfaceList[0].mappedAddr.addr[0], alpha, 
                       roi_surface->surfaceList[0].mappedAddr.addr[0], &params);

When compiled, I have errors as

gstdsexample.cpp:1017:32: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
                        params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                ^~~~~~~~
gstdsexample.cpp:1017:57: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
                        params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                                         ^~~~~~~~
gstdsexample.cpp:1017:83: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
          params->src_rect->top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                                                     ^~~~~~~~
gstdsexample.cpp:1017:110: error: ‘struct _NvBufSurfTransformCompositeBlendParams’ has no member named ‘src_rect’
 top=0; params->src_rect->left=0; params->src_rect->width=0; params->src_rect->height=0;
                                                                     ^~~~~~~~
gstdsexample.cpp:1018:102: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
 ransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha,
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1018:152: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
 ].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha,
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1019:112: error: cannot convert ‘double’ to ‘NvBufSurface*’ for argument ‘3’ to ‘NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend(NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurfTransformCompositeBlendParams*)’
                       roi_surface->surfaceList[0].mappedAddr.addr[0], &params);
                                                                              ^
Makefile:74: recipe for target 'gstdsexample.o' failed

Why no member named ‘src_rect’?
How to set parameters for NvBufSurfTransformCompositeBlend?

mchi · August 31, 2021, 8:13am

Thanks for the info and exploring the usage!

According to the usage description of NvBufSurfTransformCompositeBlend(),

the src0, src1, dst and alpha are NvBufSurface.
blend_params can be left un-initialized.

NvBufSurfTransform_ErrorNvBufSurfTransformCompositeBlend(NvBufSurface *src0, NvBufSurface *src1, NvBufSurface *alpha, NvBufSurface *dst, NvBufSurfTransformCompositeBlendParams *blend_params)

Thanks!

edit_or · August 31, 2021, 8:29am

I provide NvBufSurface, but I have errors as

gstdsexample.cpp: In function ‘GstFlowReturn gst_dsexample_transform_ip(GstBaseTransform*, GstBuffer*)’:
gstdsexample.cpp:1029:102: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                        NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha_surface->surfaceList[0].mappedAddr.addr[0],
                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1029:152: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                        NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha_surface->surfaceList[0].mappedAddr.addr[0],
                                                                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1029:202: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                       NvBufSurfTransformCompositeBlend(roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0], alpha_surface->surfaceList[0].mappedAddr.addr[0],
                                                                                                                                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1030:102: error: invalid conversion from ‘void*’ to ‘NvBufSurface*’ [-fpermissive]
                                                         roi_surface->surfaceList[0].mappedAddr.addr[0]);
                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gstdsexample.cpp:1030:103: error: too few arguments to function ‘NvBufSurfTransform_Error NvBufSurfTransformCompositeBlend(NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurface*, NvBufSurfTransformCompositeBlendParams*)’
                                                         roi_surface->surfaceList[0].mappedAddr.addr[0]);

Un-initialization gives me too few parameters error.

mchi · August 31, 2021, 8:37am

why is it still “roi_surface->surfaceList[0].mappedAddr.addr[0], color_surface->surfaceList[0].mappedAddr.addr[0]”? Shouldn’t be roi_surface ?

Topic		Replies	Views
Package the tensorRT inference results into GstBuffer and push them to the downstream DeepStream SDK	7	1293	October 17, 2022
How to draw rectangle on NvBufSurface by nvll_osd_draw_rectangles? DeepStream SDK	12	614	September 26, 2023
Cannot find the objectDetector_FastRCNN example DeepStream SDK deepstream	46	148	October 14, 2024
Uridecodebin with filesink DeepStream SDK	18	5752	October 12, 2021
questions regarding test application 4 DeepStream SDK	16	2500	October 12, 2021
Deepstream DeepStream SDK gstreamer , jetson , deepstream	14	410	July 9, 2024
To acquire a gray cv mat from nv12 nvbufsurface DeepStream SDK	13	47	September 23, 2024
Nvdspreprocess seems do process twice in ds6.2 DeepStream SDK	19	531	October 11, 2023
Segmentation fault when using nvds_obj_enc_process DeepStream SDK	9	468	April 26, 2024
Error generated while running the code after connecting the camera Jetson Xavier NX gstreamer , nvbugs	45	1220	January 2, 2024

Issue with plotting on image in deepstream-app

Related topics