Need help to convert nvbuffer to cv::mat using libargus on TX2

[jon7]

Hello, I am trying to convert nvbuffer to cv::mat on TX2 but I cannot get the proper image.
Here is my environment:
platform: TX2. platform OS: jetpack 4.5, L4T 32.5.
Argus Version: 0.97.3 (multi-process)
CSI Camera: LI-IMX577-MIPI-CS.

Basically, I am using the “oneShot” sample code provided by jetson_multimedia_api and added some lines of code to convert the nvbuffer to mat. After running the argus_oneshot application, I can see that the jpeg image saved by EGLStream::IImageJPEG has no any issues. However, the one saved by cv::imwrite is not correct.

Please see the attachment for main.cpp and CMakeLists.txt for modified oneShot sample code. You can use any diff tool to find out the changes in main.cpp. I have also included two jpg images as well. One is saved from EGLStream::IImageJPEG, the other is saved by cv::imwrite.
oneShot.zip (7.7 MB)

I cannot find anything wrong in my code, I need help to point out where my code is not correct or at least a direction to proceed.

Thanks,

Jon Liu

[ShaneCCC]

Please reference to /usr/src/jetson_multimedia_api/samples/04_video_dec_trt/video_dec_trt_main.cpp
to use NvBufferTransform

[jon7]

Hi ShaneCCC,
Thanks for your reply. Based on your suggestion, I have given it a thought again. According to this post, NvBufferTransform is done inside

iNativeBuf->createNvBuffer(iSensorMode->getResolution(),NvBufferColorFormat_ABGR32, NvBufferLayout_Pitch);

line of code. So from camera, I got the image in YUV420 format (iEGLStreamSettings->setPixelFormat(Argus::PIXEL_FMT_YCbCr_420_888);) and then I call createNvBuffer to convert it to ABGR32 with pitch layout. Then I create the cv::mat by

cv::Mat imgbuf = cv::Mat(iSensorMode->getResolution().height(), iSensorMode->getResolution().width(),CV_8UC4, pdata);

I suspect that it is either the NvBufferColorFormat or NvBufferLayout is to be adjusted or changed. Could you shed more light to this issue?

Thanks

Jon Liu

[DaneLLL]

Hi,
Please refer to this patch:
NVBuffer (FD) to opencv Mat - #6 by DaneLLL

[jon7]

Hello,
I figured out the issue. Since I used pitch layout, I have to specify the pitch as step size when creating the cv::Mat. So the to create cv::Mat, the following code should be used:

cv::Mat imgbuf = cv::Mat(iSensorMode->getResolution().height(), iSensorMode->getResolution().width(),CV_8UC4, pdata, nvbufparams.pitch[0]);

Now I have another question, this post has clearly explained how pitch layout works. But what about block linear? So in block linear layout, if I have image size WxH and if I would like to access pixel P at location (r,c), what do I do?

Thanks

Jon Liu

[DaneLLL]

Hi,
Please convert to pitch linear. Layout of block linear is not public.

[jon7]

Hi Dane,
Thanks for your help. I will use pitch linear then