Need help to convert nvbuffer to cv::mat using libargus on TX2

Hello, I am trying to convert nvbuffer to cv::mat on TX2 but I cannot get the proper image.
Here is my environment:
platform: TX2. platform OS: jetpack 4.5, L4T 32.5.
Argus Version: 0.97.3 (multi-process)
CSI Camera: LI-IMX577-MIPI-CS.

Basically, I am using the “oneShot” sample code provided by jetson_multimedia_api and added some lines of code to convert the nvbuffer to mat. After running the argus_oneshot application, I can see that the jpeg image saved by EGLStream::IImageJPEG has no any issues. However, the one saved by cv::imwrite is not correct.

Please see the attachment for main.cpp and CMakeLists.txt for modified oneShot sample code. You can use any diff tool to find out the changes in main.cpp. I have also included two jpg images as well. One is saved from EGLStream::IImageJPEG, the other is saved by cv::imwrite.
oneShot.zip (7.7 MB)

I cannot find anything wrong in my code, I need help to point out where my code is not correct or at least a direction to proceed.

Thanks,

Jon Liu

Please reference to /usr/src/jetson_multimedia_api/samples/04_video_dec_trt/video_dec_trt_main.cpp
to use NvBufferTransform

Hi ShaneCCC,
Thanks for your reply. Based on your suggestion, I have given it a thought again. According to this post, NvBufferTransform is done inside

iNativeBuf->createNvBuffer(iSensorMode->getResolution(),NvBufferColorFormat_ABGR32, NvBufferLayout_Pitch);

line of code. So from camera, I got the image in YUV420 format (iEGLStreamSettings->setPixelFormat(Argus::PIXEL_FMT_YCbCr_420_888);) and then I call createNvBuffer to convert it to ABGR32 with pitch layout. Then I create the cv::mat by

cv::Mat imgbuf = cv::Mat(iSensorMode->getResolution().height(), iSensorMode->getResolution().width(),CV_8UC4, pdata);

I suspect that it is either the NvBufferColorFormat or NvBufferLayout is to be adjusted or changed. Could you shed more light to this issue?

Thanks

Jon Liu

Hi,
Please refer to this patch:
NVBuffer (FD) to opencv Mat - #6 by DaneLLL

Hello,
I figured out the issue. Since I used pitch layout, I have to specify the pitch as step size when creating the cv::Mat. So the to create cv::Mat, the following code should be used:

cv::Mat imgbuf = cv::Mat(iSensorMode->getResolution().height(), iSensorMode->getResolution().width(),CV_8UC4, pdata, nvbufparams.pitch[0]);

Now I have another question, this post has clearly explained how pitch layout works. But what about block linear? So in block linear layout, if I have image size WxH and if I would like to access pixel P at location (r,c), what do I do?

Thanks

Jon Liu

Hi,
Please convert to pitch linear. Layout of block linear is not public.

Hi Dane,
Thanks for your help. I will use pitch linear then