How to create OpenCV cv::Mat from NvBuffer in Jetpack 5.1

[deepak.talwar1]

Hi, what is now the updated way to populate an OpenCV Matrix using Jetpack 5.1. The method described in Using OpenCV to create cv::Mat objects from images received by the Argus yuvJpeg sample program - #4 by moren1 doesn’t seem to be valid for JP5.1, as we would need to now use NvBufSurf. Additionally, the color format flags seem to have changed. I do not see the NvBufferColorFormat_ABGR32 enum anymore.

Basically, I’m looking for JP5.1 version of this snippet

// Copy the image into an OpenCV Mat object. Note that OpenCV Matrix objects store the color
// channels in the order of blue, green, and red (hence the "bgr" naming convention).
auto *iNativeBuffer = Argus::interface_cast<EGLStream::NV::IImageNativeBuffer>(iFrame->getImage());
if (!iNativeBuffer) {
    ORIGINATE_ERROR("IImageNativeBuffer not supported by Image.");
}
if (m_dmabuf == -1) {
    m_dmabuf = iNativeBuffer->createNvBuffer(
            streamResolution,
            NvBufferColorFormat_ABGR32,
            NvBufferLayout_Pitch);
    if (m_dmabuf == -1) {
        ORIGINATE_ERROR("\tFailed to create NvBuffer\n");
    }
}
if (iNativeBuffer->copyToNvBuffer(m_dmabuf) != Argus::STATUS_OK) {
    ORIGINATE_ERROR("Failed to copy frame to NvBuffer.");
}
void *pdata = nullptr;
NvBufferParams params;
NvBufferGetParams(m_dmabuf, &params);
NvBufferMemMap(m_dmabuf, plane, NvBufferMem_Read, &pdata);
NvBufferMemSyncForCpu(m_dmabuf, plane, &pdata);
cv::Mat imgbuf = cv::Mat(streamResolution.height(), streamResolution.width(), CV_8UC4, pdata, params.pitch[0]);
cv::cvtColor(imgbuf, bgr, cv::COLOR_RGBA2BGR);

Can you please provide an example that would work with the 09_argus_camera_jpeg sample application? Thank you.

[DaneLLL]

Hi,
Please refer to this patch:
NVBuffer (FD) to opencv Mat - #6 by DaneLLL

And replace NvBuffer APIs with NvBufSurfaceMap()/NvBufSurfaceUnMap()

[deepak.talwar1]

Hi @DaneLLL , I have and that is not the problem I’m currently facing. The issue is that the cv::Mat generated looks to have the wrong color channels. Could you please tell me what to replace the NvBufferColorFormat_ABGR32 and the color conversion cv::COLOR_RGBA2BGR` flags with?

I have tried a number of combinations of the following flags, and none of them seem to work. Seems like the order of the channels is no longer consistent.

NvBuffer flags tried

• NVBUF_COLOR_FORMAT_RGBA
• NVBUF_COLOR_FORMAT_BGRA
• NVBUF_COLOR_FORMAT_ARGB
• NVBUF_COLOR_FORMAT_ABGR

OpenCV color conversions tried

• COLOR_RGBA2BGR
• COLOR_BGRA2BGR

Which combinations of these would yield the same results as in the snippet in question? Thanks.

[DaneLLL]

Hi,
NvBufferColorFormat_ABGR32 is equivalent to NVBUF_COLOR_FORMAT_RGBA. Please convert to the format and map the buffer to cv::Mat

[deepak.talwar1]

Thanks @DaneLLL . The issue was that the alpha and the red channels were swapped. Thanks for your help.

[DaveYYY]

Hi,

In case you still need it, check this patch:

diff --git a/multimedia_api/ll_samples/samples/13_argus_multi_camera/main.cpp b/multimedia_api/ll_samples/samples/13_argus_multi_camera/main.cpp
index 1bd8ed9..2e83025 100644
--- a/multimedia_api/ll_samples/samples/13_argus_multi_camera/main.cpp
+++ b/multimedia_api/ll_samples/samples/13_argus_multi_camera/main.cpp
@@ -39,6 +39,8 @@
 #include <stdio.h>
 #include <stdlib.h>
 
+#include <opencv2/opencv.hpp>
+
 using namespace Argus;
 using namespace EGLStream;
 
@@ -267,7 +269,8 @@ bool ConsumerThread::threadInitialize()
     /* Allocate composited buffer */
     input_params.width = STREAM_SIZE.width();
     input_params.height = STREAM_SIZE.height();
-    input_params.colorFormat = NVBUF_COLOR_FORMAT_NV12;
+    input_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
     input_params.layout = NVBUF_LAYOUT_PITCH;
     input_params.memType = NVBUF_MEM_SURFACE_ARRAY;
     input_params.memtag = NvBufSurfaceTag_VIDEO_CONVERT;
@@ -356,8 +359,8 @@ bool ConsumerThread::threadExecute()
             {
                 batch_surf[i] = NULL;
                 m_dmabufs[i] = iNativeBuffer->createNvBuffer(iEglOutputStreams[i]->getResolution(),
-                                                          NVBUF_COLOR_FORMAT_YUV420,
-                                                          NVBUF_LAYOUT_BLOCK_LINEAR);
+                                                          NVBUF_COLOR_FORMAT_RGBA,
+                                                          NVBUF_LAYOUT_PITCH);
                 if (!m_dmabufs[i])
                     CONSUMER_PRINT("\tFailed to create NvBuffer\n");
                 if (-1 == NvBufSurfaceFromFd(m_dmabufs[i], (void**)(&batch_surf[i])))
@@ -374,7 +377,19 @@ bool ConsumerThread::threadExecute()
         {
             /* Composite multiple input to one frame */
             NvBufSurfTransformMultiInputBufCompositeBlend(batch_surf, pdstSurf, &m_compositeParam);
-            g_renderer->render(m_compositedFrame);
+            NvBufSurfaceMap(pdstSurf, -1, 0, NVBUF_MAP_READ);
+            NvBufSurfaceSyncForCpu(pdstSurf, -1, 0);
+
+            cv::Mat imgbuf = cv::Mat(STREAM_SIZE.height(),
+                                     STREAM_SIZE.width(),
+                                     CV_8UC4, pdstSurf->surfaceList->mappedAddr.addr[0]);
+            cv::Mat display_img;
+            cvtColor(imgbuf, display_img, cv::COLOR_RGBA2BGR);
+            
+            NvBufSurfaceUnMap(pdstSurf, -1, 0);
+
+            cv::imshow("img", display_img);
+            cv::waitKey(1);
         }
         else
             g_renderer->render(m_dmabufs[0]);

You also need to use opencv4 instead of opencv in Makefile:

+CPPFLAGS+=`pkg-config --cflags opencv4`
+LDFLAGS+=`pkg-config --libs opencv4`