Not able to draw line on incoming frame

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Overview:
I am using custom plugin dsexample and i want to draw line on to the frame, i have just added 1 line

i.e
cv::line(in_mat, cv::Point(100,100), cv::Point(200,100), cv::Scalar(155, 140 , 100), 3);

But OSD is not showing up and i am getting segmentation fault, below is the function which i have changed and added above mentioned line.

Note:
MyGst pipline is as follows
filesrc → h264parse ->nvv4l2decoder → nvstreammux → nvinfer → nvtracker → nvvidconv
→ mycustom → nvosd

Question:
What i am doing wrong here please help

//Memset the memory
  NvBufSurfaceMemSet (dsexample->inter_buf, 0, 0, 0);

  GST_DEBUG_OBJECT (dsexample, "Scaling and converting input buffer\n");

  // Transformation scaling+format conversion if any.
  err = NvBufSurfTransform (&ip_surf, dsexample->inter_buf, &transform_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    goto error;
  }
  // Map the buffer so that it can be accessed by CPU
  if (NvBufSurfaceMap (dsexample->inter_buf, 0, 0, NVBUF_MAP_READ) != 0){
    goto error;
  }

  // Cache the mapped data for CPU access
  NvBufSurfaceSyncForCpu (dsexample->inter_buf, 0, 0);

  // Use openCV to remove padding and convert RGBA to BGR. Can be skipped if
  // algorithm can handle padded RGBA data.
  in_mat =
      cv::Mat (dsexample->processing_height, dsexample->processing_width,
      CV_8UC4, dsexample->inter_buf->surfaceList[0].mappedAddr.addr[0],
      dsexample->inter_buf->surfaceList[0].pitch);

#if (CV_MAJOR_VERSION >= 4)
  cv::cvtColor (in_mat, *dsexample->cvmat, cv::COLOR_RGBA2BGR);
  //mycustom line that i have added
  cv::line(in_mat, cv::Point(100,100), cv::Point(200,100), cv::Scalar(155, 140 , 100), 3);
#else
  cv::cvtColor (in_mat, *dsexample->cvmat, CV_RGBA2BGR);
#endif

Error that i am getting

With tracker
Calling the plugin : krishna
Calling the plugin : krishna
Now playing: /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264

Using winsys: x11 
Opening in BLOCKING MODE 
Creating LL OSD context new
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Creating LL OSD context new
KLT Tracker Init
Segmentation fault (core dumped)

Extras:
If i am doing it in a wrong way, please guide how can i draw the line on the frame

2

Hi,
The code should be:

#if (CV_MAJOR_VERSION >= 4)
  cv::cvtColor (in_mat, *dsexample->cvmat, cv::COLOR_RGBA2BGR);
  //mycustom line that i have added
  cv::line(*dsexample->cvmat, cv::Point(100,100), cv::Point(200,100), cv::Scalar(155, 140 , 100), 3);
  cv::cvtColor (*dsexample->cvmat, in_mat, cv::COLOR_BGR2RGBA);
  NvBufSurfaceSyncForDevice (dsexample->inter_buf, 0, 0);
#else

Howeverm, the performance may not be good. We suggest you use nvdsosd.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%2520Development%2520Guide%2Fdeepstream_app_config.3.2.html%23wwpID0EQHA

3

HI @DaneLLL, I am trying to do a similar thing with rtsp streams as input and have based my code on deepstream-app with ds-example pluign enabled. The code throws segementation fault whenever I try to change BGR back to RGBA with in_mat as destination buffer,. If I replace in_mat with some other buffer, it works.

cv::cvtColor (in_mat, *dsexample->cvmat, CV_RGBA2BGR);
cv::line(*dsexample->cvmat, cv::Point(100,100), cv::Point(200,100), cv::Scalar(155, 140 , 100), 3);
cv::Mat temp_buf;
cv::cvtColor (*dsexample->cvmat, in_mat, CV_BGR2RGBA);
NvBufSurfaceSyncForDevice (dsexample->inter_buf, 0, 0);
4

Hi,
Please may the buffer to NVBUF_MAP_READ_WRITE

NvBufSurfaceMap (dsexample->inter_buf, 0, 0, NVBUF_MAP_READ_WRITE);
5

Hi @DaneLLL, Thanks for the quick reply. Really appreciate it.
Do I also need to perform a

NvBufSurfTransform

from dsexample->interbuf to ip_surf?

6

Hi,
Yes, you need to copy the buffer back to ip_surf.

7

HI @DaneLLL,
Thanks for helping me out. I am able to draw a line on the image using OpenCV functions.
Just one last question, what should be the order of these functional calls -

This -

NvBufSurfaceSyncForDevice (dsexample->inter_buf, 0, 0);
err = NvBufSurfTransform (dsexample->inter_buf, &ip_surf, &transform_params);
 if (NvBufSurfaceUnMap (dsexample->inter_buf, 0, 0)){
    goto error; 
}

or

NvBufSurfaceSyncForDevice (dsexample->inter_buf, 0, 0);
 if (NvBufSurfaceUnMap (dsexample->inter_buf, 0, 0)){
    goto error; 
} 
err = NvBufSurfTransform (dsexample->inter_buf, &ip_surf, &transform_params);

or somethingf else ?

8

Hi,
NvBufSurfTransform() has to be after NvBufSurfaceSyncForDevice(). NvBufSurfaceUnMap() is not dependent to the two functions.

9

Thanks. The problem is resolved.

10

Hi @DaneLLL, I have tried to draw rectangular bounding boxes using cv::rect using the above logic, The interval is set to 0 in both config file and spec file for primary-gie. sync is 0 and source is a single rtsp. I have observed that the boxes flicker a lot. I have disabled both tracker and osd . By flicker I mean, sometimes all 4 sides of the rectangle will be visible, sometimes only 2 , sometimes3 etc

11

Hi,
It should not happen if you call NvBufSurfaceSyncForDevice() before NvBufSurfTransform(). Could you make a patch on gstdsexample.cpp? So that we can apply it, build and run deepstream-app to reproduce the issue.

12

Hi @DaneLLL,
I will try to make a patch. In the meantime, can you give any suggestions to solve the issue. I have observed, If I draw the bounding boxes manually using OpenCV and disable the osd, the boxes flicker - let’s say I have draw a red colour bbox, sometimes 2 sides will be visible, sometimes 3 sides - the side which is not visible - in place of it a visible whitish line is present. Can I use NVOSD to draw the bboxes. Will it solve the issue? If yes, can you share a working example for the same ?

13

This is the get_converted_mat function which I am using. I have made changes only to this function.

static GstFlowReturn
get_converted_mat (GstDsExample * dsexample, NvBufSurface *input_buf, gint idx,
    NvOSD_RectParams * crop_rect_params, gdouble & ratio, gint input_width,
    gint input_height)
{
  NvBufSurfTransform_Error err;
  NvBufSurfTransformConfigParams transform_config_params;
  NvBufSurfTransformParams transform_params;
  NvBufSurfTransformRect src_rect;
  NvBufSurfTransformRect dst_rect;
  NvBufSurface ip_surf;
  cv::Mat in_mat;
  ip_surf = *input_buf;

  ip_surf.numFilled = ip_surf.batchSize = 1;
  ip_surf.surfaceList = &(input_buf->surfaceList[idx]);

  gint src_left = GST_ROUND_UP_2(crop_rect_params->left);
  gint src_top = GST_ROUND_UP_2(crop_rect_params->top);
  gint src_width = GST_ROUND_DOWN_2(crop_rect_params->width);
  gint src_height = GST_ROUND_DOWN_2(crop_rect_params->height);

  // Maintain aspect ratio
  double hdest = dsexample->processing_width * src_height / (double) src_width;
  double wdest = dsexample->processing_height * src_width / (double) src_height;
  guint dest_width, dest_height;

  if (hdest <= dsexample->processing_height) {
    dest_width = dsexample->processing_width;
    dest_height = hdest;
  } else {
    dest_width = wdest;
    dest_height = dsexample->processing_height;
  }

  // Configure transform session parameters for the transformation
  transform_config_params.compute_mode = NvBufSurfTransformCompute_Default;
  transform_config_params.gpu_id = dsexample->gpu_id;
  transform_config_params.cuda_stream = dsexample->cuda_stream;

  // Set the transform session parameters for the conversions executed in this
  // thread.
  err = NvBufSurfTransformSetSessionParams (&transform_config_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransformSetSessionParams failed with error %d", err), (NULL));
    goto error;
  }

  // Calculate scaling ratio while maintaining aspect ratio
  ratio = MIN (1.0 * dest_width/ src_width, 1.0 * dest_height / src_height);

  if ((crop_rect_params->width == 0) || (crop_rect_params->height == 0)) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("%s:crop_rect_params dimensions are zero",__func__), (NULL));
    goto error;
  }

#ifdef __aarch64__
  if (ratio <= 1.0 / 16 || ratio >= 16.0) {
    // Currently cannot scale by ratio > 16 or < 1/16 for Jetson
    goto error;
  }
#endif
  // Set the transform ROIs for source and destination
  src_rect = {(guint)src_top, (guint)src_left, (guint)src_width, (guint)src_height};
  dst_rect = {0, 0, (guint)dest_width, (guint)dest_height};

  // Set the transform parameters
  transform_params.src_rect = &src_rect;
  transform_params.dst_rect = &dst_rect;
  transform_params.transform_flag =
    NVBUFSURF_TRANSFORM_FILTER | NVBUFSURF_TRANSFORM_CROP_SRC |
      NVBUFSURF_TRANSFORM_CROP_DST;
  transform_params.transform_filter = NvBufSurfTransformInter_Default;

  //Memset the memory
  NvBufSurfaceMemSet (dsexample->inter_buf, 0, 0, 0);

  GST_DEBUG_OBJECT (dsexample, "Scaling and converting input buffer\n");

  // Transformation scaling+format conversion if any.
  err = NvBufSurfTransform (&ip_surf, dsexample->inter_buf, &transform_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    goto error;
  }
    // Map the buffer so that it can be accessed by CPU
    if (NvBufSurfaceMap (dsexample->inter_buf, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
        goto error;
    }

  // Cache the mapped data for CPU access
  NvBufSurfaceSyncForCpu (dsexample->inter_buf, 0, 0);

  // Use openCV to remove padding and convert RGBA to BGR. Can be skipped if
  // algorithm can handle padded RGBA data.
  in_mat =
      cv::Mat (dsexample->processing_height, dsexample->processing_width,
      CV_8UC4, dsexample->inter_buf->surfaceList[0].mappedAddr.addr[0],
      dsexample->inter_buf->surfaceList[0].pitch);

#if (CV_MAJOR_VERSION >= 4)
  cv::cvtColor (in_mat, *dsexample->cvmat, cv::COLOR_RGBA2BGR);
#else
  cv::cvtColor (in_mat, *dsexample->cvmat, CV_RGBA2BGR);
#endif
cv::Rect2f roi;
roi.x = 10;
roi.y = 10;
roi.width = 50;
roi.height = 50;

auto bbox_color = cv::Scalar(0, 255, 0) );
cv::rectangle(*dsexample->cvmat,roi, bbox_color, 1);
cv::putText(*dsexample->cvmat, "car", cv::Point(roi.x, roi.y-10), cv::FONT_HERSHEY_PLAIN, 1, bbox_color);	
cv::cvtColor (*dsexample->cvmat, in_mat, CV_BGR2RGBA);
   NvBufSurfaceSyncForDevice (dsexample->inter_buf, 0, 0);
  // Set the transform ROIs for source and destination
  src_rect = {(guint)0, (guint)0, (guint)dest_width, (guint)dest_height};
  dst_rect = {0, 0, (guint)dest_width, (guint)dest_height};
  // Set the transform parameters
  transform_params.src_rect = &src_rect;
  transform_params.dst_rect = &dst_rect;
  transform_params.transform_flag =
    NVBUFSURF_TRANSFORM_FILTER | NVBUFSURF_TRANSFORM_CROP_SRC |
      NVBUFSURF_TRANSFORM_CROP_DST;
  transform_params.transform_filter = NvBufSurfTransformInter_Default;

  //Memset the memory
  NvBufSurfaceMemSet (&ip_surf, 0, 0, 0);
  // Transformation scaling+format conversion if any.
  err = NvBufSurfTransform (dsexample->inter_buf, &ip_surf, &transform_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    goto error;
  }
  GST_DEBUG_OBJECT (dsexample, "Scaling and converting input buffer\n");

  if (NvBufSurfaceUnMap (dsexample->inter_buf, 0, 0)){
    goto error;
  }

#ifdef __aarch64__
  // To use the converted buffer in CUDA, create an EGLImage and then use
  // CUDA-EGL interop APIs
  if (USE_EGLIMAGE) {
    if (NvBufSurfaceMapEglImage (dsexample->inter_buf, 0) !=0 ) {
      goto error;
    }

    // dsexample->inter_buf->surfaceList[0].mappedAddr.eglImage
    // Use interop APIs cuGraphicsEGLRegisterImage and
    // cuGraphicsResourceGetMappedEglFrame to access the buffer in CUDA

    // Destroy the EGLImage
    NvBufSurfaceUnMapEglImage (dsexample->inter_buf, 0);
  }
#endif

  /* We will first convert only the Region of Interest (the entire frame or the
   * object bounding box) to RGB and then scale the converted RGB frame to
   * processing resolution. */
  return GST_FLOW_OK;

error:
  return GST_FLOW_ERROR;
}

Here, src_width and src_height are same as the destination one.

15

Hi,
Please share a complete gstdsexample.cpp. We will hit build error by applying the patch:

gstdsexample.cpp:585:6: note:   crosses initialization of ‘cv::Scalar_<double> bbox_color’
 auto bbox_color = cv::Scalar(0, 255, 0);
      ^~~~~~~~~~
gstdsexample.cpp:579:12: note:   crosses initialization of ‘cv::Rect2f roi’
 cv::Rect2f roi;

Probably need to include certain OpenCV header files. Would be helpfule for us if you share a compleate cpp file.

16

Continue in