Does VideoEncoder support RGB input?

Does VideoEncoder support RGB input?

Hi,
No, you need to convert the format to NV12(or I420). Please refer to the sample pipeline:

$ gst-launch-1.0 videotestsrc num-buffers=300 ! video/x-raw,format=BGR ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=a.mp4

Hi @DaneLLL, thanks for your reply!

I’m using the tegra multi media API, and actually the my input data is in Grayscale 8 bit format, I try to use NvBufferTransform to convert it into YUV_420M like this:

int conv_fd = 0;

{
    NvBufferCreateParams create_params = {0};
    create_params.width = width;
    create_params.height = height;
    create_params.layout = NvBufferLayout_Pitch;
    create_params.colorFormat = NvBufferColorFormat_GRAY8;
    create_params.nvbuf_tag = NvBufferTag_NONE;
    create_params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&conv_fd, &create_params) != 0) {
        return -1;
    }
}

int enc_fd = 0;

{
    NvBufferCreateParams params = {0};
    params.width = width;
    params.height = height;
    params.layout = NvBufferLayout_Pitch;
    params.colorFormat = NvBufferColorFormat_YUV420;
    params.nvbuf_tag = NvBufferTag_NONE;
    params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&enc_fd, &params) != 0) {
        return -2;
    }
}

if (0 != Raw2NvBuffer((unsigned char*)rgb_data, 0, width, height, conv_fd)) {
    return -3;
}

NvBufferTransformParams trans_params = {0};
trans_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
trans_params.transform_filter = NvBufferTransform_Filter_Smart;
if (0 != NvBufferTransform(conv_fd, enc_fd, &trans_params)) {
    return -4;
}

I save the gray scale data (rgb_data) and converted YUV data (enc_fd) like this:

FILE* dump_rgb = fopen("dump.rgb", "wb");
fwrite(rgb_data, 1, width * height, dump_rgb);
fclose(dump_rgb);

std::ofstream dump_file("dump_420.yuv");
for (uint32_t i = 0; i < 3; i++) {
    dump_dmabuf(enc_fd, i, &dump_file);
}

dump.rgb looks fine, but dump_420.yuv is covered by green color, and that’s because the U and V planes are all zero.

How could I fix the green color issue? Does NvBufferTransform support my usecase?

Hi,
Your observation is correct. We have reference code to handle this case. Please check nvbuff_do_clearchroma() in 12_camera_v4l2_cuda.

Hi @DaneLLL, I checked the sample you referred, and I know what should I do now. But I’ve another question, what should I do if I have normal RGB data?

Actually before your reply, I try to change conv_fd's param from NvBufferColorFormat_GRAY8 to NvBufferColorFormat_ARGB, and try to copy the grayscale data into conf_fd like this:

int plane = 0;
int ret = 0;
int dmabuf_fd = conf_fd;
NvBufferParams parm;
ret = NvBufferGetParams(dmabuf_fd, &parm);

void *psrc_data;

ret = NvBufferMemMap(dmabuf_fd, plane, NvBufferMem_Read_Write, &psrc_data);
if (ret == 0)
{
    unsigned int i = 0;
    NvBufferMemSyncForCpu(dmabuf_fd, plane, &psrc_data);
    //memset(psrc_data, 255, parm.height[plane]* parm.pitch[plane]);
    for (i = 0; i < parm.height[plane]; ++i)
    {
        // ARGB
        // R
        memcpy((char *)psrc_data + i * parm.pitch[plane] + parm.width[plane], 
            rgb_data + i * width,
            parm.width[plane]);
        // G
        memcpy((char *)psrc_data + i * parm.pitch[plane] + 2 * parm.width[plane], 
            rgb_data + i * width,
            parm.width[plane]);
        // B
        memcpy((char *)psrc_data + i * parm.pitch[plane] + 3 * parm.width[plane], 
            rgb_data + i * width,
            parm.width[plane]);
    }
    NvBufferMemUnMap(dmabuf_fd, plane, &psrc_data);
}

I want to make NvBufferTransform believe it’s normal RGB data, but just all color bytes have the same value, but it doesn’t work, the output YUV data is corrupted. Any suggestions?

Hi,
Not sure if the code is with correct logic. The data in NvBuffer is
R G B A R G B A ...

So you should copy 3 byte R G B, Nvbuffer pointer+1 to skip A, copy next 3 byte R G B, Nvbuffer pointer +1, and so on.

Okay thanks @DaneLLL, I’ll try that.

Hi @DaneLLL, I’m trying to convert RGBA data into YUV, below is my code:

int conv_fd = 0;

{
    NvBufferCreateParams create_params = {0};
    create_params.width = width;
    create_params.height = height;
    create_params.layout = NvBufferLayout_Pitch;
    create_params.colorFormat = NvBufferColorFormat_ARGB32;
    create_params.nvbuf_tag = NvBufferTag_NONE;
    create_params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&conv_fd, &create_params) != 0) {
        return -1;
    }
}

int enc_fd = 0;

{
    NvBufferCreateParams params = {0};
    params.width = width;
    params.height = height;
    params.layout = NvBufferLayout_Pitch;
    params.colorFormat = NvBufferColorFormat_YUV420;
    params.nvbuf_tag = NvBufferTag_NONE;
    params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&enc_fd, &params) != 0) {
        return -2;
    }
}

NvBufferParams params = {0};
NvBufferGetParams(conv_fd, &params);

uint8_t* sBaseAddr = nullptr;
NvBufferMemMap(conv_fd, 0, NvBufferMem_Read_Write, (void**)&sBaseAddr);
NvBufferMemSyncForCpu(conv_fd, 0, (void**)&sBaseAddr);

for (int i = 0; i < height; i++) {
    int dst_index = i * params.pitch[0];
    int src_index = i * width;
    for (int j = 0; j < width; j++) {
        sBaseAddr[dst_index + 0] = rgb_buffer->data()[src_index + 0];
        sBaseAddr[dst_index + 1] = rgb_buffer->data()[src_index + 1];
        sBaseAddr[dst_index + 2] = rgb_buffer->data()[src_index + 2];
        sBaseAddr[dst_index + 3] = 0xFF;
        dst_index += 4;
        src_index += 4;
    }
}

NvBufferMemSyncForDevice(conv_fd, 0, (void**)&sBaseAddr);
NvBufferMemUnMap(conv_fd, 0, (void**)&sBaseAddr);

NvBufferTransformParams trans_params = {0};
trans_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
trans_params.transform_filter = NvBufferTransform_Filter_Smart;
if (0 != NvBufferTransform(conv_fd, enc_fd, &trans_params)) {
    return -4;
}

FILE* dump_rgb = fopen("dump.rgb", "wb");
fwrite(rgb_data, 1, width * height, dump_rgb);
fclose(dump_rgb);

std::ofstream dump_file("dump_420.yuv");
for (uint32_t i = 0; i < 3; i++) {
    dump_dmabuf(enc_fd, i, &dump_file);
}

Then I inspect the dump file using http://rawpixels.net/ , the rgb data looks good (my camera only support grayscale, so even with RGBA output format, it’s still grayscale):

But the yuv data is corrupted:

Do you have any idea? Thanks!

Hi,
src_index += 4; looks not right. It should be src_index += 3;

And you may try NvBufferColorFormat_ABGR32

Hi, my source data is in rgba format, which has 4 bytes for 1 pixel.

I’ll try NvBufferColorFormat_ABGR32.

Thanks!

Hi @DaneLLL, I tried NvBufferColorFormat_ABGR32, the output YUV is still corrupted. Any idea?

Hi,
We do conversion through gstreamer commands and it works well.

$ gst-launch-1.0 videotestsrc num-buffers=1 ! video/x-raw,format=GRAY8 ! videoconvert ! video/x-raw,format=RGBA ! filesink location=test.RGBA
$ gst-launch-1.0 filesrc location= test.RGBA ! videoparse format=11 width=320 height=240 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=I420 ! filesink location=I420.yuv

The nvvidconv plugin is implemented with NvBuffer APIs, although it is not open source code. If it works in gstreamer, it should work in using NvBuffer APIs.

Please replace test.RGBA with your RGBA source and give it a try.

Hi @DaneLLL, I tried gst, but it also not working. Please download my test data here: https://imgs.piasy.com/2020-03-26-Archive.zip

After unzip, there are 3 files, dump.rgb is the RGBA data file, I420.yuv is the result of gst, dump_420.yuv is the result of NvBufferTransform.

Please help, thanks!

BTW, my gst command is: gst-launch-1.0 filesrc location=dump.rgb ! videoparse format=11 width=1920 height=1080 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=I420 ! filesink location=I420.yuv.

After inspecting the result in rawpixel.net, it seems to be resolution issue.

dump.rgb is:

I420.yuv is:

Hi,
This should be wrong:

It should be int src_index = i * width * 4;

Also somehow in dump.rgb, the alpha channel is 0xFF in some front pixels and the rest is 0x00. That’s why you see I420.yuv is whole black after the front pixels.

1 Like

Nice observation! Thanks man, you save my day!