NvBuffer to OpenCV GpuMat. Does memory is directly accessible to the device?

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

My topic is heavly based on this post

I’m currently trying to directly map a NvBuffer to a Opencv::GpuMat with NvBufferMemMap and NvBufferMemSyncForDevice on a Jetson TX2i.

I took the main from the samples available in /usr/src/jetson_multimedia_api/ and took precisely the number 07. (Video convert)
In the sample the NvBuffer is created with NvBufferCreateEx with the thread_context. I inlaided a little snippet of code in the function read_video_frame

(OpenCV Version 4.5.4).

The following is working :

void *ptr;

NvBufferMemMap(src_dma_fd, 0, NvBufferMem_Read_Write, &ptr);
NvBufferMemSyncForCpu(src_dma_fd, 0, &ptr);

cv::Mat plane_y;
static cv::Mat y_only(src_param.height[0], src_param.width[0], CV_8U);

plane_y = {src_param.height[0], // 1080
           src_param.width[0], // 1920
           CV_8U,
           ptr, // y
           src_param.pitch[0]}; // 2048
	
static cv::cuda::GpuMat cuda_plane_y(src_param.height[0], src_param.width[0], CV_8U);

cuda_plane_y.upload(plane_y);

cuda_plane_y.download(y_only);

cv::imshow("img", y_only);

cv::waitKey(0);

The following is not working :

void *ptr;

NvBufferMemMap(src_dma_fd, 0, NvBufferMem_Read_Write, &ptr);
NvBufferMemSyncForDevice(src_dma_fd, 0, &ptr);

cv::cuda::GpuMat plane_y;
static cv::Mat y_only(src_param.height[0], src_param.width[0], CV_8U);

plane_y = {src_param.height[0], // 1080
           src_param.width[0], // 1920
           CV_8U,
           ptr, // y
           src_param.pitch[0]}; // 2048

plane_y.download(y_only);

cv::imshow("img", y_only);

cv::waitKey(0);

error: (-217:Gpu API call) invalid argument in function 'download'

Changing NvBufferMemSyncForCpu does not work for me.

I’m only trying to directly map to a GpuMat to avoid memory transfer time.

What i already did to debug :

Changing my GpuMat constructor to

cv::cuda::GpuMat plane_y = cv::cuda::GpuMat(src_param.height[0], // 1080
                                            src_param.width[0], // 1920
                                            CV_8U,
                                            ptr); // 2048

Doesn’t work either.

Adding or deleting the pitch also do not change my problem.
And in a final effort, changing to a non-static y_only cv::Mat did not work.

My question is, does context have to specify something when creating a NvBuffer that the memory is shared between Device and CPU ?

Or i’m missing something important in my code to make it successful ?

Best regards.
~S.

3

Hi,
Only NvBuffer in RGBA format can be mapped to cv::Mat or cv::gpuMat. Please refer to the patches:
LibArgus EGLStream to nvivafilter - #14 by DaneLLL
NVBuffer (FD) to opencv Mat - #6 by DaneLLL

4

Hi.

Thank you for your answer. I will verify if i can map a RGBA NvBuffer to a GpuMat. But for the cv::Mat part it’s working with only one plane (Y). So actually NvBuffer can map to a cv::Mat with YUV420 format. Why does GpuMat cannot do the same operation as cv::Mat ?

~S.

5

Hi,
This would need other users to share experience. For OpenCV we are also learning from public posts. It looks like RGBA is a suitable format for hooking NvBuffer and OpenCV.

6

For NV12 format to GpuMat, you may have a look to this post:

7

Hi,

Following up my recent post, i’ve successfully mapped a ABGR32 NvBuffer to a cv::Mat. But i still struggle to map my NvBuffer to a cv::cuda::GpuMat. It looks like my HW Buffer is not allocated on the device, i’m kinda flabbergasted ysyyork managed to download it to a Cpu mat.

My question are still pending but i kinda wanna shift my focus on the jetson multimedia api context, i’m absolutly neophyte to the subject of context creation.

To make a reminder, i’m using the main inside the sample of the jetson multimedia api. (07_video_convert)

  • Do i have something to do with the context used in this sample ?
  • Can OpenCV 4.5.4 cause any trouble ? Maybe some compilation flags i’ve been missing with this library ?

Best regards.
~S.

8

Hi,

Thank you for your answer. I’m not trying or using anything related to the NV12 format. It looks like your solution is coming from a EGLImage produced by NvEGLImageFromFd. I cannot use this specific library.

Still thank you for your time.
~S.

9

Hi,
You would need to check why NvEGLImageFromFd() does not work properly in your environment. For mapping NvBuffer to cv::cuda::GpuMat, it is a must-call function.

10

Hi,
Thank you for your answer. I’m currently working in a monitorless environnement. If i can recall correctly NvEGLImageFromFd() needs to be used with an egl_display if set to NULL the default egl_display is used.

I also don’t have mesa-utils in my system.

Do you know how can i work around with no monitor wired ? I’ve been viewing some post about creating a virtual monitor but with no pratical example on how to do it.

Best regards.
~S.

11

Hi,
You may try to run this sample:
Opencv gpu mat into GStreamer without downloading to cpu - #15 by DaneLLL

If it can be run successfully, it should be fine to call the function in monitorless environment:

    egl_display = eglGetDisplay(EGL_DEFAULT_DISPLAY);
    eglInitialize(egl_display, NULL, NULL);
    eglTerminate(egl_display);
12

Hi,

Thank you for your answer. I’m sorry to disappoint again, but here’s what i’m returning with commands from EGL library.

/* Get default EGL display */
EGLDisplay egl_display;

egl_display = eglGetDisplay(EGL_DEFAULT_DISPLAY);

if (egl_display == EGL_NO_DISPLAY)
	printf("Failed to get EGL display connection\n");

/* Init EGL display connection */
if (!eglInitialize(egl_display, NULL, NULL))
	printf("Failed to initialize EGL display connection\n");

NvEglRenderer* g_renderer = NvEglRenderer::createEglRenderer("renderer0", src_param.width[0],
                                            		     src_param.height[0], 0, 0);

if (g_renderer)
	printf("Failed to create EGL renderer");

EGLImageKHR egl_image = NULL;
egl_image = NvEGLImageFromFd(egl_display,src_dma_fd);
NvDestroyEGLImage(egl_display,egl_image);

eglTerminate(egl_display);

Failed to get EGL display connection
Failed to initialize EGL display connection
[INFO] (~/Desktop/yuv422p/NvEglRenderer.cpp:110) Setting Screen width 1920 height 1080
[ERROR] (~/Desktop/yuv422p/NvEglRenderer.cpp:198) Unable to get egl display
[ERROR] (~/Desktop/yuv422p/NvEglRenderer.cpp:154) Got ERROR closing display
NvEGLImageFromFd: No EGLDisplay to create EGLImage
NvDestroyEGLImage: No EGLDisplay to destroy EGLImage

It look likes i’m also having trouble with OpenGL and drivers associated with it.

glxinfo
name of display: localhost:
X Error of failed request: BadValue (integer parameter out of range for operation)
Major opcode of failed request: 152 (GLX)
Minor opcode of failed request: 24 (X_GLXCreateNewContext)
Value in failed request: 0x0
Serial number of failed request: 41
Current serial number in output stream: 42

13

Maybe for headless you could try VirtualGL. I don’t have much experience with it, but searching this forum for that you may find some information.