Jetson multimedia API, convert UYVY to YUV420

[act-reds]

Hello,

So first of all a little context before the actual issue, so you can understand what has been done and validated.

Hardware :
Orin AGX
Sensor (camera) configured to stream UYVY (1280x720, 60fps)

Following the specific documentation to retrieve the stream:
https://docs.kernel.org/userspace-api/media/v4l/dmabuf.html

And using the NvBuffer lib to allocate memory that the DMA can access. Something like the following was done:

struct v4l2_requestbuffers req;
enum v4l2_buf_type type;

req.count = REQCOUNT;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_DMABUF;

xioctl(fd, VIDIOC_REQBUFS, &req);

buffers = (struct buffer *)calloc(req.count, sizeof(*buffers));

NvBufferCreateParams input_params = {0};

input_params.payloadType = NvBufferPayload_SurfArray;
input_params.width = 1280;
input_params.height = 720;
input_params.layout = NvBufferLayout_Pitch;

// Create the buffers
for (n_buffers = 0; n_buffers < req.count; ++n_buffers) {
	int fd_tmp;
	NvBufferParams params = {0};

	// Define pixel format of input
	input_params.colorFormat = NvBufferColorFormat_UYVY;
	
	// Define the incoming input
	input_params.nvbuf_tag = NvBufferTag_CAMERA;
	
	// Create the params
	NvBufferCreateEx(&fd_tmp, &input_params);

	fds[n_buffers] = fd_tmp;

	NvBufferGetParams(fds[n_buffers], &params);

	NvBufferMemMap(fds[n_buffers], 0, NvBufferMem_Read_Write, (void **)&buffers[n_buffers].start);
}

I removed every check to keep the code easier to follow, but everything is checked and not any error is thrown.

Then the workflow is followed and working as expected. The step allowing to retrieve the stream is validated. I have right now a picture in UYVY, that I can save in a file and then read it, displaying the expected output.

So the next step is to encode this picture in JPEG using the VIC. First of all to convert raw data in JPEG, we need this raw to be in YUV420 pixel format.
I checked the doc of Nvidia and here : Jetson Linux API Reference: Buffer Manager | NVIDIA Docs
So from the documentation, the function NvBufferTransform() : "Transforms one DMA buffer to another DMA buffer. This function can support transforms for copying, scaling, fliping, rotating, and cropping. "
The function needs :

1. src fd
2. dst fd
3. NvBufferTransformParams
   So here what I did is creating a new buffer with the expected output I want (YUV420) so I can encode it to JPEG. And then a new NvBufferTransformParams to specify what I want exactly, here you have the code :

// Buffer creation
int dst_buffer;
NvBufferCreateParams dst_params;
dst_params.payloadType = NvBufferPayload_SurfArray;
dst_params.width = 1280;
dst_params.height = 720;
dst_params.layout = NvBufferLayout_Pitch;
dst_params.colorFormat = NvBufferColorFormat_YUV420;
dst_params.nvbuf_tag = NvBufferTag_VIDEO_CONVERT;

NvBufferCreateEx(&dst_buffer, &dst_params);

NvBufferTransformParams uyvy2yuv_params;
uyvy2yuv_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
uyvy2yuv_params.transform_filter = NvBufferTransform_Filter_Smart;
uyvy2yuv_params.transform_flip = NvBufferTransform_None;

NvBufferTransform(fds[0], dst_buffer, &uyvy2yuv_params);

Also removed the checks, but not any error is thrown.
So to have an overview, the data that I can retrieve from the src (fds[0]) is in UYVY and can be saved/read as expected in a file. But when I try to save the filled buffer (dst) that should contain YUV420 data, there are 3 planes as expected, with the correct sizes (1280x720, 640x360, 640x360). When I save it in raw (YUV420) and I try to display it, I can verify that the Y plane contains the correct grayscale data, but the 2 other planes (U and V) don’t really have any information that can display the color.

Picture saving only 1 plane (Y) with grayscale data:

Picture with 3 planes, trying to interpret U, V color data:

Also when I’m trying to save the YUV420 image to JPEG with the encoding method I found in the samples of the jetson multimedia API, the output image is broken like that :

So here I am with my issue, first of all, am I doing something wrong, did I miss something ?

I’d like to mention that the code I wrote is a mix of many samples of the Jetson Multimedia API (00, 01, 02, 05, 07, 10, 12 and v4l2cuda)

I am wondering if I misunderstood something

[DaneLLL]

Hi,
Please check if you can see good camera preview when running default 12_camera_v4l2_cuda. If camera preview is good, you can then add NvJpegEncoder to encode the frames into JPEG.