Nvcompositor plugs alpha is not work and alpha plugs no work

[liuoben]

• I have the same question in the topic
  Gstreamer alpha plugin does not work

and i find the nvcompositor alpha property is not work

set “nvcompositor name=mixer sink_0::alpha=0.5”

the video fram is not will

I want remove the bkg white color,so

gst-launch-1.0
videotestsrc pattern=snow ! video/x-raw, framerate=10/1, width=1920, height=1080 !
alpha alpha=1 ! nvvidconv ! mix.sink_0

videotestsrc pattern=smpte75 ! video/x-raw, framerate=10/1, width=700, height=700 !
alpha method=custom target-g=255 target-b=255 target-r=255 ! nvvidconv ! mix.sink_1

nvcompositor name=mix sink_1::xpos=500 sink_1::ypos=100
sink_1::width=600 sink_1::height=600 sink_1::alpha=0.5 sink_0::alpha=1 ! nv3dsink \

if set "alpha method=custom target-g=255 target-b=255 target-r=255 "
the view is black
if set “alpha alpha=0.5”
the view is black too

• can you give me some suggestions? Thanks!

I find the last suggesiton like this

Hi,
Thank you for your effort to make alpha plugin works.
I have the same issue.
So, I look into commit log of GitHub - GStreamer/gst-plugins-good: ‘Good’ GStreamer plugins and helper libraries. This module has been merged into the main GStreamer repo for further development. .
After I modified as the commit below.
It’s working better.
I hope it helps.

Author: Doug Nazar nazard@nazar.ca 2019-09-09 09:43:17
Committer: Doug Nazar nazard@nazar.ca 2019-09-09 13:51:53
Parent: 0a6a62aa7652f0b5a689d4cecbc031b763c2000a (docs: Port all docstring to gtk-doc markdown)
Child: 37c619f9954045602a61daf4bceff4cd0e4f70cf (plugins: Use gst_type_mark_as_plugin_api() for all non-element plugin types)
Branches: master, remotes/origin/1.18, remotes/origin/master
Follows: 1.16.0
Precedes: 1.17.1

alpha: Fix one_over_kc calculation

On arm/aarch64, converting from float directly to unsigned int uses
a different opcode and negative numbers result in 0. Cast to
signed int first.

----------------------------- gst/alpha/gstalpha.c -----------------------------
index 867e000…4a37b01 100644
@@ -2349,7 +2349,7 @@ gst_alpha_init_params_full (GstAlpha * alpha,
tmp = MIN (tmp, 255);
alpha->accept_angle_ctg = tmp;
tmp = 1 / (kgl);

• alpha->one_over_kc = 255 * 2 * tmp - 255;

• alpha->one_over_kc = (gint) (255 * 2 * tmp - 255);
  tmp = 15 * y / kgl;
  tmp = MIN (tmp, 255);
  alpha->kfgy_scale = tmp;

but, I use the nvcompositor.

Should I update the source code of gstreamer on tx2 nx devices to this version, recompile, and then nvcompositer is ok?

[Honey_Patouceul]

@liuoben

and i find the nvcompositor alpha property is not work

1. Yes, running JP5.0.2 I also see a regression with nvcompositor no longer able to do alpha composing even with both inputs in RGBA format in NVMM memory and setting alpha/zorder (was ok in JP4):

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=RGBA,width=640,height=480 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! queue ! comp.sink_0   videotestsrc pattern=ball ! video/x-raw,width=320,height=240 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! queue ! comp.sink_1  nvcompositor name=comp sink_0::zorder=1 sink_0::alpha=1 sink_1::zorder=2 sink_1::alpha=0.1 ! nvvidconv ! nv3dsink

As workaround, you may use CPU based compositor such as:

gst-launch-1.0 -v videotestsrc ! videoconvert ! 'video/x-raw,format=RGBA,width=640,height=480' ! queue ! comp.sink_0   videotestsrc pattern=ball ! videoconvert ! 'video/x-raw,format=RGBA' ! queue ! comp.sink_1  compositor name=comp sink_0::zorder=1 sink_0::alpha=1 sink_1::zorder=2 sink_1::alpha=0.1 ! nvvidconv ! nv3dsink

2. For removing white parts, I’m not positive…alpĥa plugin may still be buggy on ARM64. The above patch doesn’t solve the issue by itself, AFAIU. Seems it is applied in gtreamer 1.16.3, and even building 1.18.6 (for newer gstreamer version it would need a newer meson version than what is available from apt, you may get newer meson version from pip but not tried), it stills remove yellow bar from smpte instead of white. Note that a newer gstreamer version may not allow using nvidia gstreamer plugins built for a lower gstreamer release.

Someone better skilled may further advise about these 2 points.

[liuoben]

Thank you!
At first, I used the compositor, but after compositor ,the width was wrong. I don’t know why .so I use nvcompositor,but the alpha and alpha is not work…

[liuoben]

I find the alpha plugs bug at here:

alpha: doesn't correctly work with black or white in custom mode (#270) ·...

The bug is no closed and alpha can remove black ,white,gray color .
and on the tx nx2,the alpha method = blue,the blue will be black.

we can find the gstalpha.c, set code like this

alpha->one_over_kc = (gint) (255 * 2 * tmp - 255);

and rebuild gst-plugins-good.

sudo cp /usr/local/lib/aarch-linux-64/gstreamer-1.0/lisbgstalpha.so to /usr/lib/aarch-linux-64/gstreamer-1.0

it will be nvcompositor alpha property is work.

but The alpha plugins it just set blue to black is not set Transparent at 1.14.5…

In my device ,the gst versiong is gst-1.14.5 ,new gst version will be not support nv* plugs, so must use the source code is really gst version!

Thanks!

[Honey_Patouceul]

Got it. Indeed with gstreamer 1.16.3 the alpha plugin custom method works for colors.
You would have to use compositor. The problem you had with wrong size may be related to your pipeline setting a source size of 700x700 but setting compositor input sink_1 as 600x600.

The following works:

gst-launch-1.0 -v videotestsrc pattern=snow ! videoconvert ! 'video/x-raw,format=RGBA,width=640,height=480' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=RGBA ! queue ! comp.sink_0   videotestsrc pattern=smpte ! videoconvert ! 'video/x-raw,format=RGBA' ! alpha method=custom target-r=0 target-g=0 target-b=255 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=RGBA ! queue ! comp.sink_1  compositor name=comp sink_0::zorder=1 sink_0::alpha=1 sink_1::zorder=2 sink_1::alpha=0.5 ! nvvidconv ! nv3dsink

But with nvcompositor the RGBA inputs seem to be rendered on black background before composing:

gst-launch-1.0 -v videotestsrc pattern=snow ! videoconvert ! 'video/x-raw,format=RGBA,width=640,height=480' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! queue ! comp.sink_0   videotestsrc pattern=smpte ! videoconvert ! 'video/x-raw,format=RGBA' ! alpha method=custom target-r=0 target-g=0 target-b=255 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! queue ! comp.sink_1  nvcompositor name=comp sink_0::zorder=1 sink_0::alpha=1 sink_1::zorder=2 sink_1::alpha=0.5 ! nvvidconv ! nv3dsink

@DaneLLL,
Am I missing something ? Or are having alpha rendered on inputs and inability to alpĥa compose known issues ?

[liuoben]

Yes !

I find alpha method=bule → videoconvert → nvcompositor is work !
alpha method=blue ->nvvidconv → nvcompositor is not work! the blue is set to black!

I think the alpha plugs no problem, the black reason may be nvvidconv → nvcompositor set alpha=0 to black!

and nvvidconv->compositor is work,but my gui size is 1024*600,and the video src width :height= 1.3:1

so the realy show video on my screen is width=780,height =600.
composition scale the video frame by used the same ratio

My compositor version is 1.14.5,is so distressed!
@DaneLLL Cane you give some advice to us?

[Honey_Patouceul]

@liuoben

I don’t think that nvvidconv is the culprit… using :

... alpha plugin -> nvvidconv as RGBA into NVMM memory -> nvvidconv as RGBA into system memory -> compositor ...  

works fine.

For your display issue with compositor, it may be related to pixel-aspect-ratio. Add verbose flag (-v) to gst-launch command so that you can get more insight about it.
You may force it to 1/1 in caps before compositor inputs (nvvidconv may also do the job such as here).
On compositor output, you may also resize to your screen resolution and add pixel-aspect-ratio=1/1 into caps.

[liuoben]

Yes!
… alpha plugin → nvvidconv as RGBA into NVMM memory → nvvidconv as RGBA into system memory → compositor …

is fine !

but alpha plugin → nvvidconv as RGBA into NVMM memory → nvvidconv as RGBA into system memory → nvcompositor has some problems

on my device:
the pipline is ok!!
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

but this is not work !!!
std::string pipeline =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! alpha method=blue "
“! queue2”
"! nvvidconv "
"! video/x-raw(memory:NVMM),format=(string)RGBA "
"! nvcompositor name = compos interpolation-method=0 sink_0::zorder=1 sink_0::alpha=0.85 "
"sink_1::alpha=1 "
"sink_1::zorder=0 "
"! nvvidconv "
“! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
“! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc”
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
“! nvvidconv”
"! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600 "
“! queue2”
“! nvvidconv”
"! compos. ";

[liuoben]

This is ok too!
std::string pipeline =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! alpha method=blue "
“! queue2”
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! nvcompositor name = compos sink_0::zorder=1 sink_0::alpha=0.85 "
"sink_1::alpha=1 "
"sink_1::zorder=0 "
"! videoconvert "
“! autovideosink”
// "! nvvidconv "
// “! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
// “! nvv4l2h264enc "
// “! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc”
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
“! nvvidconv”
"! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600 "
“! queue2”
“! videoconvert”
"! compos. ";

the error is nvvidconv->nvcompositor,but
the pipiline is pk
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

so …
…

[liuoben]

is not work
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

is work
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

just not same is memory:NVMM
@DaneLLL

[liuoben]

use glvideomixer instend nvcompositor,but this will use CPU and memory,and use cpu
use nvcompositor, the nvvidcov → nvcompositor the opcaity=0 will be black… It is so sad.
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
“! queue2”
"! alpha method=blue "
"! glvideomixer name = compos sink_0::zorder=1 sink_0::alpha=0.85 sink_1::alpha=1 sink_1::zorder=0 "
“! nvvidconv”
“! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
“! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc”
“! video/x-raw(memory:NVMM),format=(string)NV12, width=(int)1640, height=(int)1232, framerate=(fraction)25/1”
“! queue2”
“! nvvidconv”
“! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600 , pixel-aspect-ratio=1/1”
"! compos. ";

[liuoben]

Hi , I use the pipeline like this.

std::string pipeline_nvcompsositor =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! alpha method=blue "
“! queue2”
“! nvvidconv”
"! nvcompositor name = compos sink_0::zorder=1 sink_0::alpha=0.4 "
"sink_1::alpha=1 "
"sink_1::zorder=0 "
"! nvvidconv "
"! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc "
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
“! nvvidconv”
"! video/x-raw(memory:NVMM),format=(string)RGBA, width=(int)1024, height=(int)770 "
“! nvvidconv”
"! videobalance contrast=0.8 brightness=0.35 "
"! nvvidconv "
“! queue2”
"! compos. ";

because the nvvidconv->nvcompositor the black problem,i use the sink_0::alpha=0.4 ,then the brightness of the frame is very dark

so i use the videobalance contrast=0.8 brightness=0.35 ,add its brightness,The effect,Since there is a dark transparent layer in the picture, the picture looks white after adjusting the brightness.

Because i use cv::VideoWrite，write one picture to video src, and The white color in the picture is a bit like noise, do you know which gst plug-in can eliminate it ?

Thanks!

[DaneLLL]

Hi,
A quick solution is to use nvivafilter plugin and apply alpha effect to each pixel directly. Please refer to attached CUDA code.
nvsample_cudaprocess.zip (2.8 KB)

[liuoben]

I just find nvivafilter use example from generic-no-api_r2
the pipeline like this:
nvivafilter cuda-process=true customer-lib-name=libnvsample_cudaprocess.so

In my pipeline the appsrc add one BGR,the bkg is blue.

if use video/x-raw(memory:NVMM),nvvidconv → nvcompositor ,alpha it does not work, i guess in gpu the aransparency is sacrificed for computational efficiency!

I understand your suggestion,but my alpha image is appsrc from cv::VideoWrite,while i inster nvivafilter to
nvvidconv → nvcompositor for save alpha channel,

but the pipeline report error:

N:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module appsrc0 reported: Internal data stream error.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…

My pipeline like this
std::string pipeline_nvcompsositor =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
“! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600”
"! alpha method=blue "
“! queue2”
"! nvvidconv "
"! nvivafilter cuda-process=true customer-lib-name=libnvsample_cudaprocess.so "
“! video/x-raw(memory:NVMM),format=(string)RGBA”
"! nvcompositor name = compos sink_0::zorder=1 sink_0::alpha=0.24 "
"sink_1::alpha=1 "
"sink_1::zorder=0 sink_1::width=1024 sink_1::height=600 "
"! nvvidconv "
“! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
"! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc "
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
"! nvvidconv left=0 right=1640 top=136 bottom=1096 "
"! video/x-raw(memory:NVMM),format=(string)RGBA , width=(int)1024, height=(int)600 "
“! queue2”
"! nvvidconv "
“! video/x-raw(memory:NVMM),format=(string)RGBA”
"! compos. ";

• Is there something wrong with my usage? nvivafilter

[liuoben]

Basically determine is this step

! nvvidconv ! ‘video/x-raw(memory:NVMM ), format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1

lost transparency, where can see the source code of nvvidconv，and nvcompositor plugs?

I think this step ignores transparency for efficiency.

I want to confirm my idea, can you tell me where to get the source code

on my device, version of nvcompositor is 1.8.3, nvvidconv is 1.2.3

Thank you very much!!!

[DaneLLL]

Hi,
The source code is in
https://developer.nvidia.com/embedded/l4t/r35_release_v1.0/sources/public_sources.tbz2

Please take a look. You can follow README to manually build the plugins. And can customize the plugin to add the function.

[liuoben]

Ok,thank you!

[liuoben]

In gstnvvconv.c
from video/x-raw to vide/x-raw( memory:NVMM) ,code is like this

else if (space->inbuf_memtype == BUF_MEM_SW && space->outbuf_memtype == BUF_MEM_HW) {
         ..
         ..
          NvBufSurfaceCopy (&ip_surf, space->input_interbuf.isurface);

          retn = NvBufSurfTransform (space->input_interbuf.isurface, op_surf, &space->transform_params);
          if (retn != 0) {
            g_print ("%s: NvBufSurfTransform Failed \n", __func__);
            flow_ret = GST_FLOW_ERROR;
            goto done;
          }

the function NvBufSurfTransform is in /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so

can you tell me where to get the source code of libnvbufsurftransform.so,?

[DaneLLL]

Hi,
NvBufSurface APIs ate not open source.

[liuoben]

Oh .

In gstnvcompositor.c,the NvBufSurfTransformMultiInputBufCompositeBlend function exec compositor blend.

 
  nvcomp->comp_params.params.composite_blend_flag |= NVBUFSURF_TRANSFORM_COMPOSITE;
  nvcomp->comp_params.params.composite_blend_flag |= NVBUFSURF_TRANSFORM_COMPOSITE_FILTER;
 

  ret = NvBufSurfTransformMultiInputBufCompositeBlend(input_nvbuf_surfs, dst_nvbuf_surf, &nvcomp->comp_params);
  if (ret != 0) {
    GST_ERROR ("NvBufSurfTransformMultiInputBufComposite failed");
    return FALSE;
  }

The problem nvvidconv → nvcompositor use

#define GST_CAPS_FEATURE_MEMORY_NVMM “memory:NVMM”

may be exist one bug .

Or is one bug between plug-ins cohesion, either NvBufSurfTransform or NvBufSurfTransformMultiInputBufCompositeBlend !

So, i had to find other ways around the problem…