How to properly access NvBuffer color format NV12

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi,

I try to understand how to process NV12 color format.
Please, see the following code.

If I create the NvBuffer as a ABGR32, the dump of the bytes is (almost) as expected (actually, it is RGBA-8bit). However, if I change the format to NV12 the dumped Y channel is completely screwed up.

At this link, you can find a zip file containing memory dumps, sample images and the Matlab scripts I use to load the binary files:
https://drive.google.com/open?id=1PGDzl_EZFqF0-u2uRHorpw71WV_YPI4r

How to properly manage the image planes for YUV images?

Thank you.

IFrameConsumer *i_frame_consumer = interface_cast<IFrameConsumer>(m_frame_consumer);
    if (!i_frame_consumer)
        ORIGINATE_ERROR("Failed to retrieve IFrameConsumer interface\n");

    IEGLOutputStream *i_egl_output_stream = interface_cast<IEGLOutputStream>(m_output_stream);
    if (!i_egl_output_stream)
        ORIGINATE_ERROR("failed to get IEGLOutputStream interface\n");

    // acquire a frame
    UniqueObj<Frame> frame(i_frame_consumer->acquireFrame());
    IFrame *i_frame = interface_cast<IFrame>(frame);
    if (!i_frame)
        ORIGINATE_ERROR("Failed to retrieve iFrame interface\n");

    // get the IImageNativeBuffer extension interface
    NV::IImageNativeBuffer *i_native_buffer =
        interface_cast<NV::IImageNativeBuffer>(i_frame->getImage());
    if (!i_native_buffer)
        ORIGINATE_ERROR("IImageNativeBuffer not supported by Image!\n");

    // if we don't already have a buffer, create one from this image.
    // Otherwise, just blit to our buffer.
    if (m_dmabuf == -1)
    {
#if USE_NV12_FMT
        m_dmabuf = i_native_buffer->createNvBuffer(i_egl_output_stream->getResolution(),
                                                 NvBufferColorFormat_NV12,
                                                 NvBufferLayout_Pitch);
#else
        m_dmabuf = i_native_buffer->createNvBuffer(i_egl_output_stream->getResolution(),
                                                 NvBufferColorFormat_ABGR32, // actually, this is RGBA_8-8-8-8
                                                 NvBufferLayout_Pitch);
#endif
        if (m_dmabuf == -1)
            ORIGINATE_ERROR("failed to create NvBuffer\n");
    }
    else if (i_native_buffer->copyToNvBuffer(m_dmabuf) != STATUS_OK)
    {
        ORIGINATE_ERROR("failed to copy frame to NvBuffer\n");
    }

    f.number = i_frame->getNumber();
    f.timestamp = i_frame->getTime();

    Image *image = i_frame->getImage();
    if (!image)
        ORIGINATE_ERROR("failed to retrieve image\n");
    IImage *iImage = interface_cast<IImage>(image);
    if (!iImage)
        ORIGINATE_ERROR("failed to retrieve iImage\n");
    IImage2D *iImage2D = interface_cast<IImage2D>(image);
    if (!iImage2D)
        ORIGINATE_ERROR("failed to retrieve iImage2D\n");
    if(iImage->getBufferCount() == 0)
        ORIGINATE_ERROR("insane num of buffers for the image!\n");

    const Size2D<uint32_t> size = iImage2D->getSize(0);
    const uint32_t stride = iImage2D->getStride(0);

#if USE_NV12_FMT

    void *pdata_y = nullptr;
    NvBufferMemMap(m_dmabuf, 0, NvBufferMem_Read, &pdata_y);
    NvBufferMemSyncForCpu(m_dmabuf, 0, &pdata_y);

    void *pdata_uv = nullptr;
    NvBufferMemMap(m_dmabuf, 1, NvBufferMem_Read, &pdata_uv);
    NvBufferMemSyncForCpu(m_dmabuf, 1, &pdata_uv);

    std::ofstream out;
    out.open("nvbuffer_y.bin");
    out.write((const char*)pdata_y, size.height() * stride);
    out.close();
    out.open("nvbuffer_uv.bin");
    out.write((const char*)pdata_uv, (size.height() / 2) * stride);
    out.close();

    NvBufferMemUnMap(m_dmabuf, 0, &pdata_y);
    NvBufferMemUnMap(m_dmabuf, 1, &pdata_uv);

#else

    void *pdata = nullptr;
    NvBufferMemMap(m_dmabuf, 0, NvBufferMem_Read, &pdata);
    NvBufferMemSyncForCpu(m_dmabuf, 0, &pdata);

    std::ofstream out;
    out.open("nvbuffer_rgba.bin");
    out.write((const char*)pdata, size.height() * stride * 4);
    out.close();

    NvBufferMemUnMap(m_dmabuf, 0, &pdata);

#endif

memory_dump.zip (3.78 MB)

2

hello marco.carletti,

please refer to L4T Multimedia API Reference, you may check sample application to demonstrates how to use the libv4l2 video conversion component.
thanks

3

Hi Jerry,

Thank you for your reply: I’ve checked the sample you linked and it is really helpful.
However, I realized that probably my question was a bit confusing.

What I need is understanding how to read the NV12 data after the copyToNvBuffer function in order to extract the bytes of the Y, U and V channels.
Currently, I want to read the pixel values of the YUV channels, not convert the color format.

For example, when using the RGBA format, I can simply parse the byte values to extract color channels as described in the attached mat files. Since I’m assuming the policy to access the bytes is similar for all the color formats, I’ve tried reading the Y and UV planes from the NV12 format. However, it is clear there is something wrong in my procedure (see lines 62-79).

What am I missing? Is it right to consider NV12 as 2-channels data buffer (Y and UV planes)?
Is it possible to read NV12/YUV bytes similarly to what I do for RGBA? If so… How?

Thanks.

4

Hi,
NV12M is two-plane format. One Y plane and the other UV plane. If you would like to have independent U, V planes, please create NvBuffer in YUV420M.

There are APIs for accessing the buffer. You may refer to implementation of dump_dmabuf()
https://docs.nvidia.com/jetson/l4t-multimedia/group__l4t__mm__nvutils__group.html#ga3c5db44a50ef330fc9239140f2d0d35c

5

That function is exactly what I need! Now I’m able to read the memory of the NV12 color format and correctly split the Y and UV planes.
The problem was related to the wrong step value: I used “stride” instead of the actual “pitch” value.

Thank you.

1 Like