Libargus to Opencv , Error in higher resolution

[ikoc]

Hello ,

I am using libargus for getting image from mipi-csi camera then converting to opencv mat on Tx2 with jetpack 4.4.1 . It is working fine in 1920x1080 , 3840x2160 . But when i tried higher resolution like 5472x3648 converting opencv creates corrupted image . 5472x3648 → this resolution works fine while using nvjpegencoder.

I use this code for converting opencv

 IFrame *iFrame = interface_cast<IFrame>(frame);
if (!iFrame)
    ORIGINATE_ERROR("Failed to get IFrame interface");
// Get the IImageNativeBuffer extension interface.
NV::IImageNativeBuffer *iNativeBuffer =
    interface_cast<NV::IImageNativeBuffer>(iFrame->getImage());
if (!iNativeBuffer)
    ORIGINATE_ERROR("IImageNativeBuffer not supported by Image.");
if (m_dmabuf == -1)
{
    m_dmabuf = iNativeBuffer->createNvBuffer(m_resolution,
                                             NvBufferColorFormat_ABGR32,
                                             NvBufferLayout_Pitch);
    if (m_dmabuf == -1)
        ORIGINATE_ERROR("\tFailed to create NvBuffer\n");
}
else if (iNativeBuffer->copyToNvBuffer(m_dmabuf) != STATUS_OK)
{
    ORIGINATE_ERROR("Failed to copy frame to NvBuffer.");
}
void *pdata = NULL;
NvBufferMemMap(m_dmabuf, 0, NvBufferMem_Read, &pdata);
NvBufferMemSyncForCpu(m_dmabuf, 0, &pdata);
cv::Mat imgbuf = cv::Mat(m_resolution.height(), m_resolution.width(), CV_8UC4, pdata);
cv::cvtColor(imgbuf, mat, cv::COLOR_RGBA2BGR);
NvBufferMemUnMap(m_dmabuf, 0, &pdata);

This for nvjpegencoder

m_dmabuf = iNativeBuffer->createNvBuffer(m_resolution,
                                            NvBufferColorFormat_YUV420,
                                            NvBufferLayout_BlockLinear); 

What would be the solution for this problem?

[ShaneCCC]

How about configure as NvBufferLayout_BlockLinear the same with nvjpegencoder?

[ikoc]

Hello ShaneCCC, thank you for your reply,

I tried your suggestion,
m_dmabuf = iNativeBuffer->createNvBuffer(m_resolution,
NvBufferColorFormat_ABGR32,
NvBufferLayout_BlockLinear);

NvBufferLayout_Pitch _ Old one →

NvBufferLayout_BlockLinear →

[ShaneCCC]

Configure argus output as 256 alignment like 5376*3584

[ikoc]

resolution : 5376x3584
m_dmabuf = iNativeBuffer->createNvBuffer(m_resolution,
NvBufferColorFormat_ABGR32,
NvBufferLayout_BlockLinear);

result still same.

[ShaneCCC]

How do you configure the argus output size?

    Argus::Size2D<uint32_t> outputSize;
    PROPAGATE_ERROR(getOutputSize(&outputSize));

    Argus::ICaptureSession *iCaptureSession =
        Argus::interface_cast<Argus::ICaptureSession>(session);
    if (!iCaptureSession)
        ORIGINATE_ERROR("Failed to get ICaptureSession interface");

    Argus::UniqueObj<Argus::OutputStreamSettings> outputStreamSettings(
        iCaptureSession->createOutputStreamSettings(Argus::STREAM_TYPE_EGL));
    Argus::IEGLOutputStreamSettings* iEGLOutputStreamSettings =
        Argus::interface_cast<Argus::IEGLOutputStreamSettings>(outputStreamSettings);
    if (!iEGLOutputStreamSettings)
        ORIGINATE_ERROR("Failed to get IEGLOutputStreamSettings interface");

    iEGLOutputStreamSettings->setPixelFormat(m_captureYuvFormat.get());
    iEGLOutputStreamSettings->setResolution(outputSize);
    iEGLOutputStreamSettings->setEGLDisplay(Composer::getInstance().getEGLDisplay());
    iEGLOutputStreamSettings->setMetadataEnable(enableMetadata);

    Argus::UniqueObj<Argus::OutputStream> outputStream(
        iCaptureSession->createOutputStream(outputStreamSettings.get()));
    if (!outputStream)
        ORIGINATE_ERROR("Failed to create OutputStream");

[ikoc]

Actually really similar to yours.

Argus::Size2D<uint32_t> m_resolution;
m_resolution.height() = m_configuration->height;
m_resolution.width() = m_configuration->width;
printf("Camera resolution: w:%d h:%d\n", m_resolution.width(), m_resolution.height());

streamSettings = UniqueObj<OutputStreamSettings>(iCaptureSession->createOutputStreamSettings(STREAM_TYPE_EGL));
iEglStreamSettings = interface_cast<IEGLOutputStreamSettings>(streamSettings);
if (!iEglStreamSettings)
{
    ORIGINATE_ERROR("Failed to create EglOutputStreamSettings");
}
iEglStreamSettings->setPixelFormat(PIXEL_FMT_YCbCr_420_888);
//iEglStreamSettings->setMode(EGL_STREAM_MODE_MAILBOX);
iEglStreamSettings->setResolution(m_resolution);
iEglStreamSettings->setMetadataEnable(true);
m_outputStream.reset(iCaptureSession->createOutputStream(streamSettings.get()));

Output of printf : Camera resolution: w:5376 h:3584

[ShaneCCC]

Does the image normally without opencv pipeline?

[ikoc]

Yes it does, it runs with the pipeline below. Only problem is converting to cv::Mat object with resolution > 3840x2160

gst-launch-1.0
nvarguscamerasrc sensor-id=0 sensor-mode=0
! ‘video/x-raw(memory:NVMM), width=5376, height=3584, format=(string)NV12, framerate=(fraction)5/1’
! nvvidconv ! clockoverlay shaded-background=true font-desc=‘Sans 38’ ! ximagesink -e

[ShaneCCC]

The format pass to createNvBuffer() should be NvBufferColorFormat_YUV420?

[ikoc]

I tried CV_8UC4 with COLOR_RGBA2BGR variable, it gave me segmentation fault at cvtColor line.
Then i tried CV_8UC3 with COLOR_YUV2BGR_NV21 and it gave this error

terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(4.1.1) /home/nvidia/host/build_opencv/nv_opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)1; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
> Invalid number of channels in input image:
>     'VScn::contains(scn)'
> where
>     'scn' is 3

Code below:

NV::IImageNativeBuffer *iNativeBuffer =
        interface_cast<NV::IImageNativeBuffer>(iFrame->getImage());
    if (!iNativeBuffer)
        ORIGINATE_ERROR("IImageNativeBuffer not supported by Image.");

if (m_dmabuf == -1)
{
    m_dmabuf = iNativeBuffer->createNvBuffer(m_resolution,
                                             NvBufferColorFormat_YUV420,
                                             NvBufferLayout_BlockLinear);
    if (m_dmabuf == -1)
        ORIGINATE_ERROR("\tFailed to create NvBuffer\n");
}
else if (iNativeBuffer->copyToNvBuffer(m_dmabuf) != STATUS_OK)
{
    ORIGINATE_ERROR("Failed to copy frame to NvBuffer.");
}

void *pdata = NULL;
if(NvBufferMemMap(m_dmabuf, 0, NvBufferMem_Read, &pdata) == -1)
    ORIGINATE_ERROR("NvBufferMemMap failed.");

if(NvBufferMemSyncForCpu(m_dmabuf, 0, &pdata)== -1)
    ORIGINATE_ERROR("NvBufferMemSyncForCpu failed.");

//cv::Mat imgbuf = cv::Mat(m_resolution.height(), m_resolution.width(), CV_8UC4, pdata);
cv::Mat imgbuf = cv::Mat(m_resolution.height(), m_resolution.width(), CV_8UC3, pdata);

if(imgbuf.empty())
    ORIGINATE_ERROR("imgbuf empty.");

std::cout << "1" << std::endl;
//cv::cvtColor(imgbuf, mat, cv::COLOR_RGBA2BGR);
cv::cvtColor(imgbuf, mat, cv::COLOR_YUV2BGR_NV21);
std::cout << "2" << std::endl;

[ShaneCCC]

Have a try this solution.

Converting from YUV colour space to RGB using OpenCV

[ikoc]

I am not sure how but actually the first code I shared worked with 5376x3584 resolution without problem. Thank you for your support.