Subject: Significant Differences in Output Between BGRA and NV12 NvBuffer with Jetson Multimedia API
Description:
I’m working with the Jetson Multimedia API on a Jetson platform, using NvBuffers in both BGRA and NV12 formats. I’ve implemented two blocks of code to handle frames in each format, but I’m observing significant differences in the resulting images.
Here’s the code for handling BGRA and NV12:
// BGRA handling
if (frameInfoRGB->fd < 0) {
batch_surf = NULL;
frameInfoRGB->fd = iNativeBuffer->createNvBuffer(iEglOutputStream->getResolution(),
NVBUF_COLOR_FORMAT_BGRA,
NVBUF_LAYOUT_PITCH);
if (!frameInfoRGB->fd)
CONSUMER_PRINT("\tFailed to create NvBuffer\n");
if (-1 == NvBufSurfaceFromFd(frameInfoRGB->fd, (void **)(&batch_surf)))
ORIGINATE_ERROR("Cannot get NvBufSurface from fd");
}
{
if (iNativeBuffer->copyToNvBuffer(frameInfoRGB->fd) != STATUS_OK) {
ORIGINATE_ERROR("Failed to copy frame to NvBuffer.");
}
batch_surf = NULL;
if (-1 == NvBufSurfaceFromFd(frameInfoRGB->fd, (void **)(&batch_surf)))
ORIGINATE_ERROR("Cannot get NvBufSurface from fd");
cudaFrameMapper.map(batch_surf);
cv::Mat cpuMat;
cudaFrameMapper.mats[0].download(cpuMat);
std::string imgName = "auto_rgba.png";
cv::imwrite(imgName, cpuMat);
}
// NV12 handling
if (frameInfoNv12->fd < 0) {
batch_surf = NULL;
frameInfoNv12->fd = iNativeBuffer->createNvBuffer(iEglOutputStream->getResolution(),
NVBUF_COLOR_FORMAT_NV12,
NVBUF_LAYOUT_PITCH);
if (!frameInfoNv12->fd)
CONSUMER_PRINT("\tFailed to create NvBuffer\n");
if (-1 == NvBufSurfaceFromFd(frameInfoNv12->fd, (void **)(&batch_surf)))
ORIGINATE_ERROR("Cannot get NvBufSurface from fd");
}
{
if (iNativeBuffer->copyToNvBuffer(frameInfoNv12->fd) != STATUS_OK) {
ORIGINATE_ERROR("Failed to copy frame to NvBuffer.");
}
batch_surf = NULL;
if (-1 == NvBufSurfaceFromFd(frameInfoNv12->fd, (void **)(&batch_surf)))
ORIGINATE_ERROR("Cannot get NvBufSurface from fd");
cudaFrameMapperNV12.map(batch_surf);
saveNV12Planes(cudaFrameMapperNV12.mats[0], "nv12_planes");
convertNV12toBGR_NPP(cudaFrameMapperNV12.mats[0].y, cudaFrameMapperNV12.mats[0].uv, "npp_converted.png");
}
Issue:
The BGRA and NV12 code blocks produce different results. I expected the two formats to yield similar visual results after conversion, but they look significantly different.
Observations:
The BGRA block works as expected: cudaFrameMapper.mats[0].download(cpuMat); directly outputs a correct image in RGB format.
The NV12 block requires an extra conversion step with convertNV12toBGR_NPP(). Despite this, the output from the NV12 block looks off, with visible color shifts.
Questions:
Is this an expected behavior? I suspect that when it converts directly to RGBA with copyToNvBuffer, it applies some sort of processing like ISP or something, but I’m not sure if that is even remotely real.
Is there a way to get the NV12 channels to do some processing and then “manually” convert it to RGB and still get the same results as with direct RGBA conversion withcopyToNvBuffer?
Any insights or suggestions for aligning these outputs more closely would be greatly appreciated.
We are using L4T 35.6.0 and jetpack 5.1.4, is that relevant for that version?
Additionally, my data is coming from the argus camera (IMX477 sensor) then I need to get the nv12 data and use it for extra processing, I don’t want to use RGBA since it uses much more memory.
I just tested this but does not seem to make any difference.
From the discussion in the linked ticket, it seems the opposite case.
In my case, I get the raw image as NV12 (PIXEL_FMT_YCbCr_420_888) and then need to convert it to RGBA.
I also tried using the NVBUF_COLOR_FORMAT_NV12_ER buffer and then manually converting to RGBA, but the result still is different from direct copy to the RGBA buffer.
vs NV12 buffer, so I can later convert from nv12 to RGBA manually.
frameInfoNv12->fd = iNativeBuffer->createNvBuffer(iEglOutputStream->getResolution(),
NVBUF_COLOR_FORMAT_NV12, // OR NVBUF_COLOR_FORMAT_NV12_ER
NVBUF_LAYOUT_PITCH);
iNativeBuffer->copyToNvBuffer(frameInfoNv12->fd);
My conclusion is that either
the produced NV12 buffer is not accurate (there is a bug or something)
Or when copied to RGBA buffer it does extra hidden processing.
It would be extremely helpful if you could at least reject the 2nd case here. Any extra detail would be greatly appreciated.
More details about what I need
I need to do my main processing with an NV12 image, and then convert it to an RGBA image,
But the problem is that when I get the NV12 buffer and then apply the color conversion (after doing my extra processing) the resulting image has a bit more tint than the normal RGBA. (though it matches the result produced by NPP conversion).
I am using the below formula
float C = (Y - 16.0f) * 1.164f;
float D = U - 128.0f;
float E = V - 128.0f;
float R = C + 1.596f * E;
float G = C - 0.392f * D - 0.813f * E;
float B = C + 2.017f * D;
However I did some tests, and it looks like the issue is not the gamma correction
I think the gamma correction is applied for the first time when imageNativeBuffer is copied to the NV12. (iNativeBuffer->copyToNvBuffer(frameInfoNv12->fd);) and later I don’t need to apply it again,
I think the issue is in the formula I use for conversion,
Any chance you can confirm the conversion formula from NVBUF_COLOR_FORMAT_NV12 to NVBUF_COLOR_FORMAT_BGRA?
I am currently using this formula but it still looks a bit off.
float nY = (Y - 16) * (255.0f / 219.0f);
float nU = (U - 128) * (255.0f / 224.0f);
float nV = (V - 128) * (255.0f / 224.0f);
// Apply BT.601 conversion formula for limited range NV12
float R = nY + 1.402f * nV;
float G = nY - 0.344136f * nU - 0.714136f * nV;
float B = nY + 1.772f * nU;
