I have an application where I need to add a transparent overlay to a camera feed and view it on the connected display. I need to make use of hardware accelerated NVIDIA gstreamer plugins because the camera feed is 3840x2160 at 30fps.
Ultimately this transparent overlay needs to be a full GUI, but for the sake of simplicity when testing, I’m instead using a static PNG file with transparency information, loaded with multifilesrc. I want this PNG to appear on top of the camera feed, showing only the RGB data, and be able to see the pixel data from the camera feed where there are alpha components in the image.
I have tried nvcompositor, however transparency information is not retained. I can only see the PNG with a black background. I’ve made sure that both inputs to the nvcompositor are in RGBA format.
I can get this to work by avoiding the use of “memory:NVMM”, however performance is then terrible.
Similarly, the standard compositor plugin works fine, but again with awful performance.
Another approach I have tried is nvoverlaysink, making use of the overlay property. Setting the camera sink to “overlay=1” and the PNG sink to “overlay=2” results in the same behviour as nvcompositor - transparency information is not retained and I can only see the PNG with a black background.
Is there any approach that I can use to overlay the transparent image without resorting to software solutions such as “compositor” which don’t provide good enough performance?
Hi,
The nvcompositor plugin is implemented through NvBuffer APIs and it calls NvBufferComposite(). It does not implement applying alpha effect. For this use-case, you can customize nvcompositor to enable alpha effect.
Thanks for confirming that nvcompositor doesn’t implement alpha.
The linked discussion has a suggestion to “allocate NvBuffer in RGBA and add the flag NVBUFFER_BLEND”.
As far as I can tell, the “do_nvcomposite” function already does both of these suggestions (found in gstnvcompositor.c)
Are you able to be more specific about what needs to change in the nvcompositor source to enable alpha?
if (!all_yuv && (nvcomp->out_pix_fmt == NvBufferColorFormat_ABGR32)) {
nvcomp->comp_params.composite_flag |= NVBUFFER_BLEND;
GST_INFO("%d = %f", i, nvcomp->comp_params.dst_comp_rect_alpha[i]);
GST_INFO("0x%08x", nvcomp->comp_params.composite_flag);
}
I get this debug when running the pipeline
0:00:00.380176840 11230 0x559ebcade0 INFO nvcompositor gstnvcompositor.c:1309:do_nvcomposite: 1 = 0.000000
0:00:00.380232934 11230 0x559ebcade0 INFO nvcompositor gstnvcompositor.c:1310:do_nvcomposite: 0x00000007
0:00:00.431282145 11230 0x559ebcade0 INFO nvcompositor gstnvcompositor.c:1309:do_nvcomposite: 2 = 0.000000
0:00:00.431332512 11230 0x559ebcade0 INFO nvcompositor gstnvcompositor.c:1310:do_nvcomposite: 0x00000007
This confirms that NVBUFFER_BLEND is being set. It strikes me as odd that the alpha value is 0 for both inputs, despite me specifying in the pipeline for them both to be 1.0. Manually modifying this alpha value within the source code doesn’t seem to have any effect on the output.
I get the exact same debug for piplelines that use “memory:NVMM” and pipelines that do not use NVMM. However as I mention previously the pipeline without NVMM actually gives the correct result (albeit with poor framerates).
Just to note that there is already a property to set alpha values and perform blending. This already works as expected for both NVMM and non NVMM so there is no need for me to implement that. My specific issue is with the scenario where both inputs are set to alpha=1.0.
If I set both inputs to alpha=0.5 in the sourcecode, then I see the camera feed with a png overlay (as expected), however the image is dark. I assume this is because the alpha data in the png is being interpreted as black pixel data and then being blended with the camera image.
What I want to do is set both inputs to alpha=1.0 and directly overlay the png, making use of the alpha data that exists in the png source file. This works for non-NVMM, but when using NVMM memory, I just see the png file with a black background (completely covering the camera feed behind). The transparency information of the source doesn’t seem to be interpreted correctly.
I’ve included examples below. Apologies for the poor quality
[1] Alpha = 0.5, without NVMM (note the dark background) - Expected result
Hi,
The use-case is not supported. NvBufferComposite() blends each video plane in a fixed alpha value. It doesn’t handle the case that alpha value varies pixel by pixel. Would suggest use software compositor plugin.
If you would like to display text, another approach is to call NvBufferMemMap() to get CPU pointer to the buffer, and use cairo APIs to put text.
Thanks for clarifying. As mentioned previously, the software compositor plugin is not suitable since my camera source is 3840x2160 30fps which the software compositor cannot keep up with.
I need to display a dynamic Qt Widget based GUI overlay rather than just the text from my example, so cairo API’s aren’t suitable.
It sounds like there is no official way to achieve the per-pixel alpha overlay that I require while making use of the NVIDIA hardware accelerated gstreamer plugins. Is there any plan to add support for this in the future?
gst-launch-1.0 commands are attached as .sh scripts in a previous response. In case you can’t access those for some reason, I’ll include here as inline code
Hi,
We have checked this and the issue is because alpha effect is applied in nvvidconv plugin and the frames to nvcompositor do not contain the transparency. Please apply the patch to rebuild nvvidconv plugin and try again:
Thank you for providing this patch, it resolves the issue I was seeing. Makes sense that the issue was with nvvidconv now that you have pointed out the fix.
I have one remaining issue. This patch works as long as the input and the output are of the same resolution (in my case 3840x2160). However if the input source resolution differs from the output resolution (e.g. 1920x1080 input and 3840x2160 output) then the alpha (transparency) information is lost, and a black background is observed.
You can see in the patch code that the conversion does not run on any input that requires scaling. The “if” statement requires the condition:(!space->do_scaling)
Would it be possible to update the patch to consider sources that require scaling and still need to retain alpha?
If you would like to replicate the issue, I have modified your gstreamer command to scale down the png source to 1920x1080 (using nvvidconv) and then scale back up to 3840x2160 (using nvvidconv). You will see the alpha data is lost and you get a black background. Alternatively you could just use a 1920x1080 png image source.
This may not be possible since for doing scaling, it would need to call NvBufferTransform(). In this step the alpha is applied. Please scale it through videoscale plugin, or have the overlay in same resolution.
I expected that might be the case. The reality of my situation is that my overlay source is actually a windowed application with a transparent background that I am grabbing using the ximagesrc plugin. Combined with nvcompositor, this allows me to use the application as an OSD style overlay to the camera feed.
When the overlay application is 3840x2160, ximagesrc tends to drop frames since Xorg is single threaded and is maxing out one of the CPU cores. My hope was that I could drop the overlay application to 1920x1080 (this way no frames are dropped) and then scale the resolution up to 3840x2160 as part of the gstreamer pipeline before feeding it into the nvcompositor.
My testing shows that the videoscale plugin causes a drop in framerate because it’s a software based plugin so isn’t suitable.
And as mentioned above, using a source of the same resolution is not feasible either due to Xorg.
I don’t suppose there is an NVIDIA equivalent for videoscale that would offer better performance?
This worked a treat! Thank you for taking the time to provide these patches. For anyone who stumbles across this post in the future and wants to get this working, install the two attached files to the following locations on the Jetson file system
