Jetson Xavier AGX problems with GPUMat

Hi,
I’m trying to handle a gpumat from a gstreamer pipeline. My code is the following:

static GstPadProbeReturn frame_processing (GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
{
  GstBuffer *buffer=(GstBuffer *) info->data;
  GstMapInfo map={0};
  int dmabuf_fd=0;
  gst_buffer_map (buffer, &map, GST_MAP_READ);
  ExtractFdFromNvBuffer((void *)map.data, &dmabuf_fd);
  
  {  
    EGLImageKHR egl_image;
    egl_image=NvEGLImageFromFd(egl_display, dmabuf_fd);
    CUresult status;
    CUeglFrame egl_frame;
    CUgraphicsResource p_resource=NULL;
    cudaFree(0);
    status=cuGraphicsEGLRegisterImage(&p_resource, egl_image, CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE);

    status=cuGraphicsResourceGetMappedEglFrame(&egl_frame, p_resource, 0, 0);
    status=cuCtxSynchronize();
    
    cv::cuda::GpuMat d_mat(H, W, CV_8UC4, egl_frame.frame.pPitch[0]);
    
    cuda::cvtColor(d_mat, d_mat,cv::COLOR_BGRA2GRAY);
    
    cv::Mat img;
    d_mat.download(img);
    circle(img, cv::Point(500,500), 300, cv::Scalar(0,255,0),-1);
    d_mat.upload(img);
   
    status=cuCtxSynchronize();
    status=cuGraphicsUnregisterResource(p_resource);
  }

  gst_buffer_unmap(buffer, &map);
  return GST_PAD_PROBE_OK;
}

And this code is calling with:

gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER, frame_processing, NULL, NULL);

I know the code doesn’t make a lot of sense, but I want to understand how that works.
My problem is that with this code, I never see the circle in the image and the image always has 4 color channels.
But if I remove

cuda::cvtColor(d_mat, d_mat,cv::COLOR_BGRA2GRAY);

I always see the circle in the image.
I don’t understand what is the problem with cvtColor.
Does anyone have any idea?

Thanks in advance

cvtColor would reallocate dmat buffer, so dmat data would no longer be the same address than egl_frame.

You would use another GpuMat with 1 channel for gray, so that dmat is untouched.

You may add traces for verifying that dmat.data is the same as egl_frame.

That might make sense, but I don’t understand how to make d_mat go gray and circle appear so I can see it in the output stream.

Regarding to add traces I understand that the egl_frame and d_mat.data should have the same address??
After creating d_mat I have put the following couts:

cout << "address egl_frame: " << &egl_frame.frame.pPitch[0] << endl;
cout << "address d_mat : " << &d_mat.data << endl;

And I see different addresses, I’m not sure about if I’m doing it right.

Thanks

That might be far from an optimal path, but you may try:

    ...
    cv::cuda::GpuMat d_mat_BGRA(H, W, CV_8UC4, egl_frame.frame.pPitch[0]);
    cv::cuda::GpuMat d_mat_GRAY(H, W, CV_8UC1);
    cuda::cvtColor(d_mat_BGRA, d_mat_GRAY, cv::COLOR_BGRA2GRAY);
    cuda::cvtColor(d_mat_GRAY, d_mat_BGRA, cv::COLOR_GRAY2BGRA);
    //Now d_mat_BGRA is color mat with monochrome image
    // Note this doesn't preserve alpha channel

    cv::Mat img;
    d_mat_BGRA.download(img);
    circle(img, cv::Point(500,500), 300, cv::Scalar(0,255,0),-1);
    d_mat_BGRA.upload(img);

    cout << "address egl_frame: " << &egl_frame.frame.pPitch[0] << endl;
    cout << "address d_mat : " << &d_mat_BGRA.data << endl;
    ...

I understand now, thanks for the quick reply!!

I have more doubts about this topic, when I put the code with the couts to see the addresses, I see that they are different.
I expected that with that code the memory addresses were the same.

Furthermore, I don’t understand why it wouldn’t work by putting at the end of that code:

d_mat_BGRA = d_mat_GRAY.clone();

instead of:

cuda::cvtColor(d_mat_GRAY, d_mat_BGRA, cv::COLOR_GRAY2BGRA);

Sorry I missed there was en error in your code and pasted it as such. Use instead:

cout << "address egl_frame: " << egl_frame.frame.pPitch[0] << endl;
cout << "address d_mat : " << d_mat_BGRA.data << endl;

About clone, since d_mat_GRAY is a one channel mat, it would make d_Mat_BGRA to become a one channel Mat as well, so d_Mat_BGRA would have a new buffer allocated at a different address.

With that I obtain a segmentation fault with:

d_mat_BGRA.data

I see… You would just add a cast to void*:

cout << "address d_mat : " << (void*)d_mat_BGRA.data << endl;

In fact, in my case I’m just testing both addresses and issue a message if different:

if (d_mat_BGRA.data != egl_frame.frame.pPitch[0])
   std::cerr << "Error: reallocated buffer for d_Mat_RGBA" << std::endl;