Unable to composite multiple NvBufSurfaces in JetPack 5.x

[smilerlee]

Hi, we are migrating the usage of MMAPI to JetPack 5.x from 4.x, according to the migration guide, NvBufferComposite is replaced by NvBufSurfTransformMultiInputBufComposite, but this new API cannot be found anywhere in nvbufsurftransform.h and/or libnvbufsurftransform.so; only the NvBufSurfTransformComposite API can be found, which requires all input buffers in a single batched-NvBufSurface, apparently not what we need.

Is there any way we can use the new NvBufSurfTransformMultiInputBufComposite API or a similar one? Thanks for your help.

[DaneLLL]

Hi,
The function is demonstrated in the sample:

/usr/src/jetson_multimedia_api/samples/13_argus_multi_camera/

Please take a look.

[smilerlee]

Should I use NvBufSurfTransformMultiInputBufCompositeBlend instead of the doc-noted NvBufSurfTransformMultiInputBufComposite ? Alright, I’ll give it a try. Thanks.

[smilerlee]

Thanks, it worked.

Yet I have another question: the area not covered by any dst_comp_rect will always filled with the black color, not the color specified in NvBufSurfTransformCompositeBlendParamsEx.params.color_bg (like NvBufferCompositeParams.composite_bgcolor worked in legacy API), any idea for this?

[DaneLLL]

Hi,
Please have all surfaces in RGBA and set the flag NVBUFSURF_TRANSFORM_BLEND. See if applying the setting works.

[smilerlee]

Verified the NVBUFSURF_TRANSFORM_COMPOSITE | NVBUFSURF_TRANSFORM_BLEND flag, didn’t work either, still black bg.

Meanwhile, our input buffers came from the video decoder, of which the format is something like NV12. Transforming every single frame of every camera/stream into BGRA might impact the system overall performance. Is there any other approaches?

[DaneLLL]

Hi,
Certain fucntions are supported in RGBA only so need to convert the surfaces into RGBA first. Fr optimal performance of conversion, please refer to the page to run hardware converter(VIC engine) at maximum clock:
VPI - Vision Programming Interface: Performance Benchmark

[smilerlee]

Yes, all inputs/output are in RGBA, with flag NVBUFSURF_TRANSFORM_COMPOSITE | NVBUFSURF_TRANSFORM_BLEND, didn’t work either.

[smilerlee]

@DaneLLL Here attached a demo code:

#include "nvbufsurftransform.h"
#include <assert.h>
#include <string.h>
#include <opencv2/imgcodecs.hpp>

int main() {
    NvBufSurfaceAllocateParams create_params;
    memset(&create_params, 0, sizeof(create_params));
    create_params.params.width = 100;
    create_params.params.height = 100;
    create_params.params.memType = NVBUF_MEM_SURFACE_ARRAY;
    create_params.params.layout = NVBUF_LAYOUT_PITCH;
    create_params.params.colorFormat = NVBUF_COLOR_FORMAT_BGRA;
    create_params.memtag = NvBufSurfaceTag_VIDEO_CONVERT;
    NvBufSurface *src0, *src1, *dst;
    int ret = NvBufSurfaceAllocate(&src0, 1, &create_params);
    assert(ret == 0);
    ret = NvBufSurfaceAllocate(&src1, 1, &create_params);
    assert(ret == 0);

    create_params.params.width = 300;
    create_params.params.height = 300;
    ret = NvBufSurfaceAllocate(&dst, 1, &create_params);
    assert(ret == 0);

    NvBufSurfTransformCompositeBlendParamsEx comp_params;
    NvBufSurfTransform_ColorParams color_bg;
    NvBufSurfTransformRect src_comp_rect[2];
    NvBufSurfTransformRect dst_comp_rect[2];
    memset(&comp_params, 0, sizeof(comp_params));
    comp_params.params.composite_blend_flag = NVBUFSURF_TRANSFORM_COMPOSITE | NVBUFSURF_TRANSFORM_BLEND;
    comp_params.params.input_buf_count = 2;
    color_bg.red = 0.5;
    color_bg.green = 0.5;
    color_bg.blue = 0.5;
    color_bg.alpha = 1;
    comp_params.params.color_bg = &color_bg;
    src_comp_rect[0].left = 0;
    src_comp_rect[0].top = 0;
    src_comp_rect[0].width = 100;
    src_comp_rect[0].height = 100;
    src_comp_rect[1].left = 0;
    src_comp_rect[1].top = 0;
    src_comp_rect[1].width = 100;
    src_comp_rect[1].height = 100;
    comp_params.src_comp_rect = src_comp_rect;
    dst_comp_rect[0].left = 0;
    dst_comp_rect[0].top = 0;
    dst_comp_rect[0].width = 100;
    dst_comp_rect[0].height = 100;
    dst_comp_rect[1].left = 100;
    dst_comp_rect[1].top = 100;
    dst_comp_rect[1].width = 100;
    dst_comp_rect[1].height = 100;
    comp_params.dst_comp_rect = dst_comp_rect;
    // float alpha[2] = {1, 1};
    // comp_params.alpha = alpha;

    ret = NvBufSurfaceMemSet(src0, 0, 0, 255);
    assert(ret == 0);
    src0->numFilled = 1;
    ret = NvBufSurfaceMemSet(src1, 0, 0, 255);
    assert(ret == 0);
    src1->numFilled = 1;
    NvBufSurface *srcs[2] = {src0, src1};
    ret = NvBufSurfTransformMultiInputBufCompositeBlend(srcs, dst, &comp_params);
    assert(ret == 0);

    ret = NvBufSurfaceMap(dst, 0, 0, NVBUF_MAP_READ);
    assert(ret == 0);
    auto &params = dst->surfaceList[0];
    cv::Mat mat = cv::Mat(300, 300, CV_8UC4, params.mappedAddr.addr[0], params.pitch);
    cv::imwrite("output.jpg", mat);
    ret = NvBufSurfaceUnMap(dst, 0, 0);
    assert(ret == 0);

    NvBufSurfaceDestroy(src0);
    NvBufSurfaceDestroy(src1);
    NvBufSurfaceDestroy(dst);
    return 0;
}

Output:

The black pixels are expected to be gray.

[DaneLLL]

Hi,
Thaks for the test sample. would be great if you can share the make command so that we can built it and give it a try.

[smilerlee]

g++ -std=c++11 -I/usr/include/opencv4 -I/usr/src/jetson_multimedia_api/include main.cc -L/usr/lib/aarch64-linux-gnu/tegra -lnvbufsurface -lnvbufsurftransform -lopencv_core -lopencv_imgcodecs -o main

[smilerlee]

@DaneLLL Hi,
is there any progress on this one?

[DaneLLL]

Hi,
We are checking this. May have this be supported in next Jetpack 5.1.2.

[DaneLLL]

Hi,
Please try Jetpack 5.1.2.