How to convert GstNvInferFrame rgb memory to an opencv image

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi I can see there is this variable in GstNvInferFrame given below.

/** Pointer to the converted frame memory. This memory contains the frame

  • converted to RGB/RGBA and scaled to network resolution. This memory is
  • given to NvDsInferContext as input for pre-processing and inferencing. */
    gpointer converted_frame_ptr;

which is coming from GstNvInferMemory

/** Vector of pointer to individual frame memories in the batch memory */
std::vector<void *> frame_memory_ptrs;

and same is being used in gstnvinfer.cpp like this to access.

memory->frame_memory_ptrs[idx]

i want to know if there is a way i can convert this to opencv format or some kind usable format to process. i cannot find any more document for this particular variable.

please let me know.

2

Hi,
We have sample code in dsexample plugin. Please look at

deepstream_sdk_v4.0.2_jetson\sources\gst-plugins\gst-dsexample

It is in get_converted_mat()/gstdsexample.cpp

Furthermore, you may refer to this post for using cv::cuda APIs:
https://devtalk.nvidia.com/default/topic/1047620/deepstream-sdk/how-to-create-opencv-gpumat-from-nvstream-/post/5397368/#5397368

3

Ok i have tried to the same. and i am getting weird image. i am attaching one sample.

i am using below code in the function convert_batch_and_push_to_input_thread after NvBufSurfTransform.
Not sure what is going wrong.

static gboolean
convert_batch_and_push_to_input_thread (GstNvInfer *nvinfer,
    GstNvInferBatch *batch, GstNvInferMemory *mem)
{
  NvBufSurfTransform_Error err;
  std::string nvtx_str;
  cv::Mat in_mat;

  /* Set the transform session parameters for the conversions executed in this
   * thread. */
  err = NvBufSurfTransformSetSessionParams (&nvinfer->transform_config_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (nvinfer, STREAM, FAILED,
        ("NvBufSurfTransformSetSessionParams failed with error %d", err), (NULL));
    return FALSE;
  }

  nvtxEventAttributes_t eventAttrib = {0};
  eventAttrib.version = NVTX_VERSION;
  eventAttrib.size = NVTX_EVENT_ATTRIB_STRUCT_SIZE;
  eventAttrib.colorType = NVTX_COLOR_ARGB;
  eventAttrib.color = 0xFFFF0000;
  eventAttrib.messageType = NVTX_MESSAGE_TYPE_ASCII;
  nvtx_str = "convert_buf batch_num=" + std::to_string(nvinfer->current_batch_num);
  eventAttrib.message.ascii = nvtx_str.c_str();

  nvtxDomainRangePushEx(nvinfer->nvtx_domain, &eventAttrib);

  /* Batched tranformation. */
  err = NvBufSurfTransform (&nvinfer->tmp_surf, mem->surf,
            &nvinfer->transform_params);

  std::cout<<nvinfer->transform_params.dst_rect->width<<" "<<nvinfer->transform_params.dst_rect->height;

  nvtxDomainRangePop (nvinfer->nvtx_domain);

  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (nvinfer, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    return FALSE;
  }

  // Map the buffer so that it can be accessed by CPU
//  if (NvBufSurfaceMap (mem->surf, 0, 0, NVBUF_MAP_READ) != 0){
//      return FALSE;
//  }

//   Cache the mapped data for CPU access
  NvBufSurfaceSyncForCpu (mem->surf, 0, 0);

  in_mat = cv::Mat (1080, 1920,CV_8UC4, &mem->surf->surfaceList[0].mappedAddr.addr[0],mem->surf->surfaceList[0].pitch);
  cv::cvtColor (in_mat, in_mat, CV_RGB2BGR);
  cv::imwrite("out1.jpg",in_mat);

g_mutex_lock (&nvinfer->process_lock);
  /* Push the batch info structure in the processing queue and notify the output
   * thread that a new batch has been queued. */
  g_queue_push_tail (nvinfer->input_queue, batch);
  g_cond_broadcast (&nvinfer->process_cond);
  g_mutex_unlock (&nvinfer->process_lock);
  return TRUE;
}

out1.jpg
out1.jpg1920×1080 1.8 MB

4

Hi,
There are samples of saving to JPG. Please refer to 2, 5 in FAQ