Does VideoEncoder support RGB input?

Piasy · March 21, 2020, 4:28am

DaneLLL · March 23, 2020, 12:08am

Hi,
No, you need to convert the format to NV12(or I420). Please refer to the sample pipeline:

$ gst-launch-1.0 videotestsrc num-buffers=300 ! video/x-raw,format=BGR ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=a.mp4

Piasy · March 25, 2020, 2:54am

Hi @DaneLLL, thanks for your reply!

I’m using the tegra multi media API, and actually the my input data is in Grayscale 8 bit format, I try to use NvBufferTransform to convert it into YUV_420M like this:

int conv_fd = 0;

{
    NvBufferCreateParams create_params = {0};
    create_params.width = width;
    create_params.height = height;
    create_params.layout = NvBufferLayout_Pitch;
    create_params.colorFormat = NvBufferColorFormat_GRAY8;
    create_params.nvbuf_tag = NvBufferTag_NONE;
    create_params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&conv_fd, &create_params) != 0) {
        return -1;
    }
}

int enc_fd = 0;

{
    NvBufferCreateParams params = {0};
    params.width = width;
    params.height = height;
    params.layout = NvBufferLayout_Pitch;
    params.colorFormat = NvBufferColorFormat_YUV420;
    params.nvbuf_tag = NvBufferTag_NONE;
    params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&enc_fd, &params) != 0) {
        return -2;
    }
}

if (0 != Raw2NvBuffer((unsigned char*)rgb_data, 0, width, height, conv_fd)) {
    return -3;
}

NvBufferTransformParams trans_params = {0};
trans_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
trans_params.transform_filter = NvBufferTransform_Filter_Smart;
if (0 != NvBufferTransform(conv_fd, enc_fd, &trans_params)) {
    return -4;
}

I save the gray scale data (rgb_data) and converted YUV data (enc_fd) like this:

FILE* dump_rgb = fopen("dump.rgb", "wb");
fwrite(rgb_data, 1, width * height, dump_rgb);
fclose(dump_rgb);

std::ofstream dump_file("dump_420.yuv");
for (uint32_t i = 0; i < 3; i++) {
    dump_dmabuf(enc_fd, i, &dump_file);
}

dump.rgb looks fine, but dump_420.yuv is covered by green color, and that’s because the U and V planes are all zero.

How could I fix the green color issue? Does NvBufferTransform support my usecase?

DaneLLL · March 25, 2020, 4:32am

Hi,
Your observation is correct. We have reference code to handle this case. Please check nvbuff_do_clearchroma() in 12_camera_v4l2_cuda.

Piasy · March 25, 2020, 5:53am

Hi @DaneLLL, I checked the sample you referred, and I know what should I do now. But I’ve another question, what should I do if I have normal RGB data?

Actually before your reply, I try to change conv_fd’s param from NvBufferColorFormat_GRAY8 to NvBufferColorFormat_ARGB, and try to copy the grayscale data into conf_fd like this:

int plane = 0;
int ret = 0;
int dmabuf_fd = conf_fd;
NvBufferParams parm;
ret = NvBufferGetParams(dmabuf_fd, &parm);

void *psrc_data;

ret = NvBufferMemMap(dmabuf_fd, plane, NvBufferMem_Read_Write, &psrc_data);
if (ret == 0)
{
    unsigned int i = 0;
    NvBufferMemSyncForCpu(dmabuf_fd, plane, &psrc_data);
    //memset(psrc_data, 255, parm.height[plane]* parm.pitch[plane]);
    for (i = 0; i < parm.height[plane]; ++i)
    {
        // ARGB
        // R
        memcpy((char *)psrc_data + i * parm.pitch[plane] + parm.width[plane], 
            rgb_data + i * width,
            parm.width[plane]);
        // G
        memcpy((char *)psrc_data + i * parm.pitch[plane] + 2 * parm.width[plane], 
            rgb_data + i * width,
            parm.width[plane]);
        // B
        memcpy((char *)psrc_data + i * parm.pitch[plane] + 3 * parm.width[plane], 
            rgb_data + i * width,
            parm.width[plane]);
    }
    NvBufferMemUnMap(dmabuf_fd, plane, &psrc_data);
}

I want to make NvBufferTransform believe it’s normal RGB data, but just all color bytes have the same value, but it doesn’t work, the output YUV data is corrupted. Any suggestions?

DaneLLL · March 25, 2020, 7:59am

Hi,
Not sure if the code is with correct logic. The data in NvBuffer is
R G B A R G B A ...

So you should copy 3 byte R G B, Nvbuffer pointer+1 to skip A, copy next 3 byte R G B, Nvbuffer pointer +1, and so on.

Piasy · March 25, 2020, 8:01am

Okay thanks @DaneLLL, I’ll try that.

Piasy · March 26, 2020, 2:59am

Hi @DaneLLL, I’m trying to convert RGBA data into YUV, below is my code:

int conv_fd = 0;

{
    NvBufferCreateParams create_params = {0};
    create_params.width = width;
    create_params.height = height;
    create_params.layout = NvBufferLayout_Pitch;
    create_params.colorFormat = NvBufferColorFormat_ARGB32;
    create_params.nvbuf_tag = NvBufferTag_NONE;
    create_params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&conv_fd, &create_params) != 0) {
        return -1;
    }
}

int enc_fd = 0;

{
    NvBufferCreateParams params = {0};
    params.width = width;
    params.height = height;
    params.layout = NvBufferLayout_Pitch;
    params.colorFormat = NvBufferColorFormat_YUV420;
    params.nvbuf_tag = NvBufferTag_NONE;
    params.payloadType = NvBufferPayload_SurfArray;
    if (NvBufferCreateEx(&enc_fd, &params) != 0) {
        return -2;
    }
}

NvBufferParams params = {0};
NvBufferGetParams(conv_fd, &params);

uint8_t* sBaseAddr = nullptr;
NvBufferMemMap(conv_fd, 0, NvBufferMem_Read_Write, (void**)&sBaseAddr);
NvBufferMemSyncForCpu(conv_fd, 0, (void**)&sBaseAddr);

for (int i = 0; i < height; i++) {
    int dst_index = i * params.pitch[0];
    int src_index = i * width;
    for (int j = 0; j < width; j++) {
        sBaseAddr[dst_index + 0] = rgb_buffer->data()[src_index + 0];
        sBaseAddr[dst_index + 1] = rgb_buffer->data()[src_index + 1];
        sBaseAddr[dst_index + 2] = rgb_buffer->data()[src_index + 2];
        sBaseAddr[dst_index + 3] = 0xFF;
        dst_index += 4;
        src_index += 4;
    }
}

NvBufferMemSyncForDevice(conv_fd, 0, (void**)&sBaseAddr);
NvBufferMemUnMap(conv_fd, 0, (void**)&sBaseAddr);

NvBufferTransformParams trans_params = {0};
trans_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
trans_params.transform_filter = NvBufferTransform_Filter_Smart;
if (0 != NvBufferTransform(conv_fd, enc_fd, &trans_params)) {
    return -4;
}

FILE* dump_rgb = fopen("dump.rgb", "wb");
fwrite(rgb_data, 1, width * height, dump_rgb);
fclose(dump_rgb);

std::ofstream dump_file("dump_420.yuv");
for (uint32_t i = 0; i < 3; i++) {
    dump_dmabuf(enc_fd, i, &dump_file);
}

Then I inspect the dump file using http://rawpixels.net/ , the rgb data looks good (my camera only support grayscale, so even with RGBA output format, it’s still grayscale):

But the yuv data is corrupted:

Do you have any idea? Thanks!

DaneLLL · March 26, 2020, 3:33am

Hi,
src_index += 4; looks not right. It should be src_index += 3;

And you may try NvBufferColorFormat_ABGR32

Piasy · March 26, 2020, 3:34am

Hi, my source data is in rgba format, which has 4 bytes for 1 pixel.

I’ll try NvBufferColorFormat_ABGR32.

Thanks!

Piasy · March 26, 2020, 4:25am

Hi @DaneLLL, I tried NvBufferColorFormat_ABGR32, the output YUV is still corrupted. Any idea?

DaneLLL · March 26, 2020, 6:59am

Hi,
We do conversion through gstreamer commands and it works well.

$ gst-launch-1.0 videotestsrc num-buffers=1 ! video/x-raw,format=GRAY8 ! videoconvert ! video/x-raw,format=RGBA ! filesink location=test.RGBA
$ gst-launch-1.0 filesrc location= test.RGBA ! videoparse format=11 width=320 height=240 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=I420 ! filesink location=I420.yuv

The nvvidconv plugin is implemented with NvBuffer APIs, although it is not open source code. If it works in gstreamer, it should work in using NvBuffer APIs.

Please replace test.RGBA with your RGBA source and give it a try.

Piasy · March 26, 2020, 7:41am

Hi @DaneLLL, I tried gst, but it also not working. Please download my test data here: https://imgs.piasy.com/2020-03-26-Archive.zip

After unzip, there are 3 files, dump.rgb is the RGBA data file, I420.yuv is the result of gst, dump_420.yuv is the result of NvBufferTransform.

Please help, thanks!

Piasy · March 26, 2020, 7:46am

BTW, my gst command is: gst-launch-1.0 filesrc location=dump.rgb ! videoparse format=11 width=1920 height=1080 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=I420 ! filesink location=I420.yuv.

After inspecting the result in rawpixel.net, it seems to be resolution issue.

dump.rgb is:

I420.yuv is:

DaneLLL · March 26, 2020, 8:22am

Hi,
This should be wrong:

It should be int src_index = i * width * 4;

Also somehow in dump.rgb, the alpha channel is 0xFF in some front pixels and the rest is 0x00. That’s why you see I420.yuv is whole black after the front pixels.

Piasy · March 26, 2020, 8:54am

Nice observation! Thanks man, you save my day!