Reading R,G,B values for specific pixel from NvBufSurface

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform - Jetson
• DeepStream Version - 5.1
• TensorRT Version - 7.1.3
• NVIDIA GPU Driver Version - cuda10.2
• Reading data from NvBufSurface
• Requirement details

I’m currently trying to access the RGB values of a single pixel located at a predetermined location in a frame of video.My current approach to accomplish this is to create a cv::Mat from NvBufSurface . This approach is derived from the source code in gstdsexample.cpp , i.e.

struct _GstDsExample
{
  GstBaseTransform base_trans;

  /*
  CODE 
  */

  // OpenCV mat containing RGB data
  cv::Mat *cvmat;

  /*
  CODE 
  */
};

/**
 * Scale the entire frame to the processing resolution maintaining aspect ratio.
 * Or crop and scale objects to the processing resolution maintaining the aspect
 * ratio. Remove the padding required by hardware and convert from RGBA to RGB
 * using openCV. These steps can be skipped if the algorithm can work with
 * padded data and/or can work with RGBA.
 */
static GstFlowReturn
get_converted_mat (GstDsExample * dsexample, NvBufSurface *input_buf, gint idx,
    NvOSD_RectParams * crop_rect_params, gdouble & ratio, gint input_width,
    gint input_height){
  /*
  CODE
  */
  //Memset the memory
  NvBufSurfaceMemSet (nvbuf, 0, 0, 0);

  GST_DEBUG_OBJECT (eventdetector, "Scaling and converting input buffer\n");

  // Transformation scaling+format conversion if any.
  err = NvBufSurfTransform (&ipSurf, nvbuf, &transformParams);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (eventdetector, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    return false;
  }
  // Map the buffer so that it can be accessed by CPU
  if (NvBufSurfaceMap (nvbuf, 0, 0, NVBUF_MAP_READ) != 0){
    return false;
  }

  // Cache the mapped data for CPU access
  NvBufSurfaceSyncForCpu (nvbuf, 0, 0);
  /* Use openCV to remove padding and convert RGBA to BGR. Can be skipped if
  * algorithm can handle padded RGBA data. */
  in_mat =
    cv::Mat (dsexample->processing_height, dsexample->processing_width,
    CV_8UC4, dsexample->inter_buf->surfaceList[0].mappedAddr.addr[0],
    dsexample->inter_buf->surfaceList[0].pitch);

  cv::cvtColor (in_mat, *dsexample->cvmat, CV_RGBA2BGR);  

  /*
  CODE
  */
}

However, it was observed that when using this approach, there was a significant drop in performance i.e., fps dropped by nearly 15. On further investigation I was thinking operations like cv::cvtColor(), NvBufSurfaceSyncForCpu() were causing the fps drop. In my application as I will not be using entire converted matrix but accessing the R,G,B values of specific pixel let say (50,50) for example. Is there any different approach to access those R,G,B,A values from input_buf for that specific pixel by skipping some of these operations and create R,G,B values ?

3

All video buffers transferred inside DeepStream pipeline are in GPU memory but not system memory, to copy the data from GPU memory to the system memory as cv::Mat and process on such memory will be very inefficient. So it is not recommended to do any video(pixels) processing with openCV if the performance is important to you.

4

Thanks Fiona for the reply. This information was really helpful. But is there any efficient approach to

  • Read R,G,B values of just a single pixel from GPU memory before copying it to system memory ( or )
  • Copy the single pixel information from gpu memory to system memory instead of copying the entire image.

I tried the following methods to read R,G,B values from but didn’t get very in terms of improving fps

  1. Access R,G,B values from inmat without converting RGBA to BGR image
  2. Commenting the below statement i.e., used to Cache the mapped data for CPU access. I can access R,G,B values from inmat even after commenting this statement. Is commenting this a good approach ?
    NvBufSurfaceSyncForCpu (nvbuf, 0, 0);
5

Yes. You can read or copy with CUDA. CUDA Toolkit Documentation 12.0 Update 1 (nvidia.com)

There is no HW acceleration in opencv.

6

Thanks Fiona for pointing to Cuda documentation.After referring to the documentation here, I was able to read the NVBufSurface from GPU memory using cv::cuda::GpuMat . When using this approach I faced the following error:

what(): OpenCV(4.1.1) /home/nvidia/host/build_opencv/nv_opencv/modules/core/include/opencv2/core/private.cuda.hpp:107: error: (-216:No CUDA support) The library is compiled without CUDA support in function 'throw_no_cuda'

It seems this approach requires OpenCV with GPU support. Rebuilding OpenCV with GPU support is not viable option for my application, so it would be really helpful if you could shed some additional light on which Cuda APIs I should look into as the documentation is quite extensive.

Are there any particular functions/data types you could point me towards?

7

To use CUDA based Opencv is the most easy way. CUDA APIs are low level GPU related APIs, there is no single API to handle your case. You may need to learn about CUDA first. Or you can consult CUDA expert. There is also CUDA forum Latest CUDA/CUDA Programming and Performance topics - NVIDIA Developer Forums

8

Can you tell us what kind of operation would you do with this pixel?

9

To check if the pixel color is red based on those R,G,B values.

10

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

If it is so, you can just read the R, G, B value from the NvBufSurface. There is sample of how to get video data buffer address and layout from NvBufSurface. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.