Nvcompositor plugs alpha is not work and alpha plugs no work

Yes !

I find alpha method=bule → videoconvert → nvcompositor is work !
alpha method=blue ->nvvidconv → nvcompositor is not work! the blue is set to black!

I think the alpha plugs no problem, the black reason may be nvvidconv → nvcompositor set alpha=0 to black!

and nvvidconv->compositor is work,but my gui size is 1024*600,and the video src width :height= 1.3:1

so the realy show video on my screen is width=780,height =600.
composition scale the video frame by used the same ratio

My compositor version is 1.14.5,is so distressed!
@DaneLLL Cane you give some advice to us?

@liuoben

I don’t think that nvvidconv is the culprit… using :

... alpha plugin -> nvvidconv as RGBA into NVMM memory -> nvvidconv as RGBA into system memory -> compositor ...  

works fine.

For your display issue with compositor, it may be related to pixel-aspect-ratio. Add verbose flag (-v) to gst-launch command so that you can get more insight about it.
You may force it to 1/1 in caps before compositor inputs (nvvidconv may also do the job such as here).
On compositor output, you may also resize to your screen resolution and add pixel-aspect-ratio=1/1 into caps.

Yes!
… alpha plugin → nvvidconv as RGBA into NVMM memory → nvvidconv as RGBA into system memory → compositor …

is fine !

but alpha plugin → nvvidconv as RGBA into NVMM memory → nvvidconv as RGBA into system memory → nvcompositor has some problems

on my device:
the pipline is ok!!
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

but this is not work !!!
std::string pipeline =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! alpha method=blue "
“! queue2”
"! nvvidconv "
"! video/x-raw(memory:NVMM),format=(string)RGBA "
"! nvcompositor name = compos interpolation-method=0 sink_0::zorder=1 sink_0::alpha=0.85 "
"sink_1::alpha=1 "
"sink_1::zorder=0 "
"! nvvidconv "
“! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
“! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc”
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
“! nvvidconv”
"! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600 "
“! queue2”
“! nvvidconv”
"! compos. ";

This is ok too!
std::string pipeline =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! alpha method=blue "
“! queue2”
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! nvcompositor name = compos sink_0::zorder=1 sink_0::alpha=0.85 "
"sink_1::alpha=1 "
"sink_1::zorder=0 "
"! videoconvert "
“! autovideosink”
// "! nvvidconv "
// “! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
// “! nvv4l2h264enc "
// “! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc”
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
“! nvvidconv”
"! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600 "
“! queue2”
“! videoconvert”
"! compos. ";

the error is nvvidconv->nvcompositor,but
the pipiline is pk
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

so …

is not work
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

is work
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

just not same is memory:NVMM
@DaneLLL

use glvideomixer instend nvcompositor,but this will use CPU and memory,and use cpu
use nvcompositor, the nvvidcov → nvcompositor the opcaity=0 will be black… It is so sad.
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
“! queue2”
"! alpha method=blue "
"! glvideomixer name = compos sink_0::zorder=1 sink_0::alpha=0.85 sink_1::alpha=1 sink_1::zorder=0 "
“! nvvidconv”
“! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
“! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc”
“! video/x-raw(memory:NVMM),format=(string)NV12, width=(int)1640, height=(int)1232, framerate=(fraction)25/1”
“! queue2”
“! nvvidconv”
“! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600 , pixel-aspect-ratio=1/1”
"! compos. ";

Hi , I use the pipeline like this.

std::string pipeline_nvcompsositor =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
"! video/x-raw,format=(string)RGBA "
"! alpha method=blue "
“! queue2”
“! nvvidconv”
"! nvcompositor name = compos sink_0::zorder=1 sink_0::alpha=0.4 "
"sink_1::alpha=1 "
"sink_1::zorder=0 "
"! nvvidconv "
"! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc "
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
“! nvvidconv”
"! video/x-raw(memory:NVMM),format=(string)RGBA, width=(int)1024, height=(int)770 "
“! nvvidconv”
"! videobalance contrast=0.8 brightness=0.35 "
"! nvvidconv "
“! queue2”
"! compos. ";

because the nvvidconv->nvcompositor the black problem,i use the sink_0::alpha=0.4 ,then the brightness of the frame is very dark

so i use the videobalance contrast=0.8 brightness=0.35 ,add its brightness,The effect,Since there is a dark transparent layer in the picture, the picture looks white after adjusting the brightness.

Because i use cv::VideoWrite,write one picture to video src, and The white color in the picture is a bit like noise, do you know which gst plug-in can eliminate it ?

Thanks!

Hi,
A quick solution is to use nvivafilter plugin and apply alpha effect to each pixel directly. Please refer to attached CUDA code.
nvsample_cudaprocess.zip (2.8 KB)

I just find nvivafilter use example from generic-no-api_r2
the pipeline like this:
nvivafilter cuda-process=true customer-lib-name=libnvsample_cudaprocess.so

In my pipeline the appsrc add one BGR,the bkg is blue.

if use video/x-raw(memory:NVMM),nvvidconv → nvcompositor ,alpha it does not work, i guess in gpu the aransparency is sacrificed for computational efficiency!

I understand your suggestion,but my alpha image is appsrc from cv::VideoWrite,while i inster nvivafilter to
nvvidconv → nvcompositor for save alpha channel,

but the pipeline report error:

N:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module appsrc0 reported: Internal data stream error.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…

My pipeline like this
std::string pipeline_nvcompsositor =
"appsrc "
"! video/x-raw, format=BGR "
"! videoconvert "
“! video/x-raw,format=(string)RGBA, width=(int)1024, height=(int)600”
"! alpha method=blue "
“! queue2”
"! nvvidconv "
"! nvivafilter cuda-process=true customer-lib-name=libnvsample_cudaprocess.so "
“! video/x-raw(memory:NVMM),format=(string)RGBA”
"! nvcompositor name = compos sink_0::zorder=1 sink_0::alpha=0.24 "
"sink_1::alpha=1 "
"sink_1::zorder=0 sink_1::width=1024 sink_1::height=600 "
"! nvvidconv "
“! video/x-raw(memory:NVMM), format=(string)I420, width=(int)1024, height=(int)600”
"! nvv4l2h264enc "
“! rtspclientsink location=rtsp://192.168.20.99:8554/my_pipeline”
" nvarguscamerasrc "
“! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1640, height=(int)1232,framerate=(fraction)25/1”
"! nvvidconv left=0 right=1640 top=136 bottom=1096 "
"! video/x-raw(memory:NVMM),format=(string)RGBA , width=(int)1024, height=(int)600 "
“! queue2”
"! nvvidconv "
“! video/x-raw(memory:NVMM),format=(string)RGBA”
"! compos. ";

  • Is there something wrong with my usage? nvivafilter

Basically determine is this step

! nvvidconv ! ‘video/x-raw(memory:NVMM ), format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1

lost transparency, where can see the source code of nvvidconv,and nvcompositor plugs?

I think this step ignores transparency for efficiency.

I want to confirm my idea, can you tell me where to get the source code

on my device, version of nvcompositor is 1.8.3, nvvidconv is 1.2.3

Thank you very much!!!

Hi,
The source code is in
https://developer.nvidia.com/embedded/l4t/r35_release_v1.0/sources/public_sources.tbz2

Please take a look. You can follow README to manually build the plugins. And can customize the plugin to add the function.

Ok,thank you!

In gstnvvconv.c
from video/x-raw to vide/x-raw( memory:NVMM) ,code is like this

else if (space->inbuf_memtype == BUF_MEM_SW && space->outbuf_memtype == BUF_MEM_HW) {
         ..
         ..
          NvBufSurfaceCopy (&ip_surf, space->input_interbuf.isurface);

          retn = NvBufSurfTransform (space->input_interbuf.isurface, op_surf, &space->transform_params);
          if (retn != 0) {
            g_print ("%s: NvBufSurfTransform Failed \n", __func__);
            flow_ret = GST_FLOW_ERROR;
            goto done;
          }

the function NvBufSurfTransform is in /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so

can you tell me where to get the source code of libnvbufsurftransform.so,?

Hi,
NvBufSurface APIs ate not open source.

Oh .

In gstnvcompositor.c,the NvBufSurfTransformMultiInputBufCompositeBlend function exec compositor blend.


 
  nvcomp->comp_params.params.composite_blend_flag |= NVBUFSURF_TRANSFORM_COMPOSITE;
  nvcomp->comp_params.params.composite_blend_flag |= NVBUFSURF_TRANSFORM_COMPOSITE_FILTER;
 

  ret = NvBufSurfTransformMultiInputBufCompositeBlend(input_nvbuf_surfs, dst_nvbuf_surf, &nvcomp->comp_params);
  if (ret != 0) {
    GST_ERROR ("NvBufSurfTransformMultiInputBufComposite failed");
    return FALSE;
  }

The problem nvvidconv → nvcompositor use

#define GST_CAPS_FEATURE_MEMORY_NVMM “memory:NVMM”

may be exist one bug .

Or is one bug between plug-ins cohesion, either NvBufSurfTransform or NvBufSurfTransformMultiInputBufCompositeBlend !

So, i had to find other ways around the problem…

I think the nvcompositor blend mutli img should be like this.

Mat src = imread("C:/test1.jpg"); /*图片 */
	cout << "type src = " << src.channels() << endl;
	Mat src1 = imread("C:/fire-element.png", IMREAD_UNCHANGED);
	cout << "type = "<<src1.channels()<<endl;
	for (int i = 0; i != src.rows; ++i) {
		for (int j = 0; j != src1.cols; ++j) {
			Vec3b scalar1 = src.at<Vec3b>(i, j);
			Vec4b scalar2 = src1.at<Vec4b>(i, j);
			Vec3b scalar3;
			float dbAlpha = 1;
			for (int k = 0; k < 3; k++)
			{
				scalar3.val[k] = scalar1.val[k] * (1 - dbAlpha) + scalar2.val[k] * dbAlpha;
			}
			if (scalar2[3] != 0) {
                   src.at<Vec3b>(i, j) = scalar3;
  //if the foreground Image 's alpha =0, show the background rgb value,else the 
//background color while be black,just show foreground img with black background!!!
			}
		}
	}

nvivafilter may work as alpha plugin works, but nvcompositor would not be able to process it.
Attached a nvivafilter:
nviva-alpha-test.tgz (4.4 KB)
just doing thresholding from hue to alpha.

tar zxvf nviva-alpha-test.tgz
cd alpha-test

# You would edit Makefile and tune for your case (opencv install path...)

make

# in my case I used opencv-4.6.0 installed into /usr/local/opencv-4.6.0, so I first added (this may not be required for your case if you have a standard opencv intall, just for reference):
export LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64:/usr/local/opencv-4.6.0/lib:

It works with compositor:

gst-launch-1.0 videotestsrc pattern=snow ! video/x-raw,width=1280,height=720,framerate=30/1 ! queue ! comp.sink_0    videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvivafilter cuda-process=1 customer-lib-name=./lib-gst-custom-opencv_cudaprocess.so ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! queue ! comp.sink_1  compositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=1  sink_1::xpos=0 sink_1::ypos=0 sink_1::zorder=2 ! videoconvert ! xvimagesink

but doesn’t work with nvcompositor:

gst-launch-1.0 videotestsrc pattern=snow ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! queue ! comp.sink_0    videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvivafilter cuda-process=1 customer-lib-name=./lib-gst-custom-opencv_cudaprocess.so ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! queue ! comp.sink_1  nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=1  sink_1::xpos=0 sink_1::ypos=0 sink_1::zorder=2 ! nvvidconv ! xvimagesink

Hi,
Alpha blending is not supported in nvcompositor plugin. Please take Honey Patouceul’s suggestion.

Thank Honey Patouceul’s active help on this.

is not work
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

is work
gst-launch-1.0 videotestsrc pattern=snow ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_0 videotestsrc pattern=smpte75 ! alpha method=green ! nvvidconv ! ‘video/x-raw, format=(string)RGBA’ ! mixer.sink_1 nvcompositor name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink

just not same is memory:NVMM,

  • To be precise, transparency merging is not supported in Nvmm,but I think a small change could be made to this, which would actually improve efficiency.

When the top image opacity is zero, just retain the original value of the bottom image .

for (int i = 0; i != space_mat_.rows; ++i) {
            for (int j = 0; j != space_mat_.cols; ++j) {
                if (space_mat_.at<cv::Vec4b>(i, j)[3] != 0) {
                    cv::Vec3b scalar1 = src.at<cv::Vec3b>(i, j);
                    cv::Vec4b scalar2 = space_mat_.at<cv::Vec4b>(i, j);
                    float     dbAlpha = 0.68;
                    for (int k = 0; k != 3; ++k) {
                        scalar1.val[k] = scalar1.val[k] * (1 - dbAlpha) + scalar2.val[k] * dbAlpha;
                    }
                    src.at<cv::Vec3b>(i, j) = scalar1;
                    // if the foreground Image 's alpha =0, show the background rgb value,else the
                    // background color while be black,just show foreground img with black background!!!
                }
            }
        }

I change rtsp push_stream plan,use cv::VideoCapture → blend by code like up → cv::VideoWriter.
@Honey_Patouceul
Thanks for everyone !

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.