Mergin/Blending sources in custom plugin

[user96777]

Hi, I’m new to this community.

It is possible to work on more than 1 camera in the custom GStreamer plugin?
For example, I’ve got 2 cameras and I want to merge/blend them into 1 output video.

I started by reading frames, doing some transformation in the custom plugin, and what next? I want to merge/blend all sources into 1 output video. I need to put these streams into the queue or there is a plugin that can help me in this?

Thanks!

[DaneLLL]

Hi,
For th euse-case, you can use nvcompositor plugin. Please refer to this gst-launch-1.0 command:
Problem with nvcompositor + nvarguscamerasrc - #13 by ShaneCCC

[user96777]

I’ve seen and tested this plugin, but it works like a grid. What if I want to combine images in different shapes? For example, from the source images, I cut triangles (using a mask) and then merge these 2 triangles into 1 image?

I’ve functions that perform this type of transformation (masking and merging), but I don’t know how to translate it into a plugin. How to get frames from several sources and return one final image.

This is just an example

[DaneLLL]

Hi,
Merging/blending triangle sources are not supported by hardware converter. One possible solution is to implement CUDA code to use GPU. The NVIDIA plugins are open source and put in the package:
https://developer.nvidia.com/embedded/linux-tegra
L4T Driver Package (BSP) Sources

You can download the source code and check nvcompositor plugin. The plugin are implemented through NvBuffer APIs. The APIs are defined in

/usr/src/jetson_multimedia_api/include/nvbuf_utils.h

With function calls in the psuedo code you can get CUDA pointer and then implement the merging/blending CUDA code:

    //CUDA postprocess
    {
        EGLImageKHR egl_image;
        egl_image = NvEGLImageFromFd(egl_display, dmabuf_fd);
        CUresult status;
        CUeglFrame eglFrame;
        CUgraphicsResource pResource = NULL;
        cudaFree(0);
        status = cuGraphicsEGLRegisterImage(&pResource,
                    egl_image,
                    CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE);
        if (status != CUDA_SUCCESS)
        {
            printf("cuGraphicsEGLRegisterImage failed: %d \n",status);
        }
        status = cuGraphicsResourceGetMappedEglFrame(&eglFrame, pResource, 0, 0);
        status = cuCtxSynchronize();

        // CUDA code here; data pointer is eglFrame.frame.pPitch[0]

        status = cuCtxSynchronize();
        status = cuGraphicsUnregisterResource(pResource);
        NvDestroyEGLImage(egl_display, egl_image);
    }