No Chroma Keying with Gstreamer Alpha Plugin

As also written in this TX2 thread chroma keying does not work.
https://devtalk.nvidia.com/default/topic/1047323/jetson-tx2/gstreamer-alpha-plugin-does-not-work/post/5395725/#5395725

Via chroma keying, e.g. blue can be set as the fully transparent colour and should work with this code:

gst-launch-1.0 \
 videotestsrc pattern=snow ! video/x-raw, framerate=10/1, width=1920, height=1080 ! \
 alpha alpha=1 ! nvvidconv ! mix.sink_0 \
 \
 videotestsrc pattern=smpte75 ! video/x-raw, framerate=10/1, width=700, height=700 ! \
<b> alpha method=blue</b> ! nvvidconv ! mix.sink_1 \
 \
 nvcompositor name=mix sink_1::xpos=500 sink_1::ypos=100 \
 sink_1::width=600 sink_1::height=600 sink_1::alpha=0.5 sink_0::alpha=1 ! nv3dsink \

However, Jetson nano gives me an image with a black region instead of snow pattern, see screenshots. Transparency works fine, but that is not chroma keying. Finally, only the black (former green/blue) bar must be fully transparent.
Similar images where shown from rary in post #3 and #5. Rary wrote in #1 in his thread that his code works fine on another Linux machine.

I checked a lot of possibilities:
both alpha alpha=0…1
target-b=255
sink_0::zorder=0 sink_1::zorder=1
nvoverlaysink
nvvidconv ! nvegltransform ! nveglglessink

In my opinion it is a bug. So it would be very nice if someone would inform the developers.
I’ m using jetson nano with jetson-nano-sd-r32.2.1.zip
Thank you for your support.
Best regards,
Wilhelm


Hi,
We do not support the function in nvcompositor. Please use SW compositor plugin.

$ gst-launch-1.0  videotestsrc pattern=snow ! video/x-raw, framerate=10/1, width=1920, height=1080 !  alpha alpha=1 ! mix.sink_0   videotestsrc pattern=smpte75 ! video/x-raw, framerate=10/1, width=700, height=700 ! alpha method=blue ! mix.sink_1   compositor name=mix sink_1::xpos=500 sink_1::ypos=100  sink_1::width=600 sink_1::height=600 sink_1::alpha=0.5 sink_0::alpha=1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvoverlaysink

Hi DaneLLL,
thank you, but your code gives me again an image like the screenshots in #1. The black bar must be fully transparent.
rary used the plugin videomixer in his post #1 and that doesn’t work too, so nvcompositor is not the problem. Please see his post #1 from February 2019.

In my opinion it is a bug in the alpha plugin implementation of the jetson family.
Do you have the possibility to test the code in #2?
A screenshot from your result would be very nice.
Thank you for your support.
Best regards,
Wilhelm

I’m now pretty sure it is an alpha plugin problem.
Rary wrote in #7 → nvivafilter (takes composited image and alpha blending) …that works for rary…

[url]https://devtalk.nvidia.com/default/topic/1047323/jetson-tx2/gstreamer-alpha-plugin-does-not-work/1[/url]

Best regards,
Wilhelm

Hi,
We are able to see different output result in running the two pipelines on Jetson Nano:

$ gst-launch-1.0  videotestsrc pattern=snow ! video/x-raw, framerate=10/1, width=1920, height=1080 !  alpha alpha=1 ! mix.sink_0   videotestsrc pattern=smpte75 ! video/x-raw, framerate=10/1, width=700, height=700 ! alpha <b>method=blue</b> ! mix.sink_1   compositor name=mix sink_1::xpos=500 sink_1::ypos=100  sink_1::width=600 sink_1::height=600 sink_1::alpha=0.5 sink_0::alpha=1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvoverlaysink
$ gst-launch-1.0  videotestsrc pattern=snow ! video/x-raw, framerate=10/1, width=1920, height=1080 !  alpha alpha=1 ! mix.sink_0   videotestsrc pattern=smpte75 ! video/x-raw, framerate=10/1, width=700, height=700 ! alpha ! mix.sink_1   compositor name=mix sink_1::xpos=500 sink_1::ypos=100  sink_1::width=600 sink_1::height=600 sink_1::alpha=0.5 sink_0::alpha=1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvoverlaysink

alpha and compositor are native gstreamer plugins and should work fine. nvcompositor plugins is NVIDIA plugin to use hardware engine VIC. However, it does not support chroma keying. Suggest you use software compositor plugins for chroma keying.

Dear DaneLLL,
I have tested all pipelines since #3 with native gstreamer plugins. You wrote it should work fine, but it doesen’t work fine. Rary (see Link in #1) confirms, that alpha chroma keying does not works. On another linux machine it works for him.

Of course I see different outputs too, one output with transparency and one output with a (transparent (depending on alpha)) black bar instead of blue. This black bar should be fully transparent. That is the issue.

I think, it is a bug in the alpha implementation. Please forward the code from #2 and the pictures from #1 to the developers or to a colleague.
It can not be that a problem known since February is not solvable. It should also be in the interest of Nvidia that bugs are fixed to make a good product even better.
Thank you in advance,
best regards,
Wilhelm

Hi WiSi-Testpilot,

We will investigate this issue and update to you.
Thanks!

Thank you very much,
best regards,
Wilhelm

I have confirmed that the jetson (nano) implementation of gstreamer has a bug.
For test purpose I have installed the gstreamer on Windows7 and took the pipeline from rary #1, see link in my #1. The only thing I had to change was d3dvideosink instead of xvimagesink.

With the windows version the former green bar is fully transparent, as it should be, the jetson nano however gives wrongly a black bar.

Best regards,
Wilhelm


This is not an answer, but as workaround you would try custom method of alpha plugin.
Remove green bar:

gst-launch-1.0 -ev videotestsrc pattern=snow num-buffers=10 ! video/x-raw, framerate=10/1, width=1280, height=720 ! alpha alpha=1 ! mix.sink_0   videotestsrc pattern=smpte75 num-buffers=10 ! video/x-raw, framerate=10/1, width=640, height=480 ! alpha method=custom <b>target-g=190</b> target-b=0 target-r=0 ! mix.sink_1   videomixer name=mix sink_0::zorder=0 sink_1::zorder=1 num-buffers=10 ! video/x-raw, width=1280, height=720 ! videoconvert ! video/x-raw, width=1280, height=720,frame-aspect-ratio=4/3 ! xvimagesink window-width=1280 window-height=720

Remove dark blue bar:

gst-launch-1.0 -ev videotestsrc pattern=snow num-buffers=10 ! video/x-raw, framerate=10/1, width=1280, height=720 ! alpha alpha=1 ! mix.sink_0   videotestsrc pattern=smpte75 num-buffers=10 ! video/x-raw, framerate=10/1, width=640, height=480 ! alpha method=custom target-g=0 <b>target-b=192</b> target-r=0 ! mix.sink_1   videomixer name=mix sink_0::zorder=0 sink_1::zorder=1 num-buffers=10 ! video/x-raw, width=1280, height=720 ! videoconvert ! video/x-raw, width=1280, height=720,frame-aspect-ratio=4/3 ! xvimagesink window-width=1280 window-height=720

Remove red bar:

gst-launch-1.0 -ev videotestsrc pattern=snow num-buffers=10 ! video/x-raw, framerate=10/1, width=1280, height=720 ! alpha alpha=1 ! mix.sink_0   videotestsrc pattern=smpte75 num-buffers=10 ! video/x-raw, framerate=10/1, width=640, height=480 ! alpha method=custom target-g=0 target-b=0 <b>target-r=192</b> ! mix.sink_1   videomixer name=mix sink_0::zorder=0 sink_1::zorder=1 num-buffers=10 ! video/x-raw, width=1280, height=720 ! videoconvert ! video/x-raw, width=1280, height=720,frame-aspect-ratio=4/3 ! xvimagesink window-width=1280 window-height=720

That being said, there are indeed some differences from x86 (expecting values as 255), not sure how far alpha plugin has been tested on arm64 or if it’s L4T-related.

Note this was tested on a TX2 with R28.2, it may be different on Nano and R32.

Hi Honey_Patouceul,
thank you for your tests. I also tested the method=custom settings, see images. Just I have repeated it with your value of 192 for red. Again, only with Win7 it works for me.
In reality I will take method=custom anyway. In the image below I took blue and green to replace the green grass. For details you may have a look to nano projects.
https://devtalk.nvidia.com/default/topic/1064908/jetson-projects/jetson-nano-on-a-drone-multicopter/
Best regards,
Wilhelm

(The alpha of the smpte75 testimage was not set to 1.0, so the bars are noisy on the nano screenshots)




It is unclear from these links what were your commands.
Did you try exactly my commands ? What was the result there ?

[EDIT: just seen your attachments now…I’ve been a bit fast. You may try various values around for your release, but I’d suggest you try my exact pipeline first.]

Hi, Honey_Patouceul,
we made a step forward.
Your exact pipeline 1 with g=190 works, see image. (I have tested only 1.)
However if I modify a bit (g=192) the black bar appears again.
If I add red r=190 to switch off yellow, the black bar is back again, see image.

However this information may be important for Nvidia.
Thank you again,
Best regards,
Wilhelm


For switching the yellow bar off, these values work:External Media.

Hi, Honey_Patouceul,
thank you, that is very nice, but that is no solution for the real application.
In this post:
[url]https://devtalk.nvidia.com/default/topic/1047323/jetson-tx2/gstreamer-alpha-plugin-does-not-work/1[/url]
Rary wrote, that he worked around the problem with nvivafilter, see #7.
I tried it, see #11, but without success.

It would be very nice if you tried to get # 11 up and running. Every value for r, g and b must be possible.
Best regards,
Wilhelm

Hi,
Thank you for giving the information.

compositor plugin is native gstreamer plugin and looks like it gives different handling to x86, apple and other embedded platforms. In the source code( Index of /src/gst-plugins-bad ), it has DISABLE_ORC which gives different outputs. We are doing further clarification on this.

If you need a solution on Jetson platforms, please check Honey_Patouceul’s suggestion.

Thank you for doing further clarification.
Compositor plugin is not used in Honey_Patouceul’s pipeline, but I guess his suggestions would work with compositor and even nvcompositor too.
NV-plugins are important for me, because of the low lag. Otherwise I would try R-Pi4.

Honey_Patouceul’s suggestions are nice, but for the real application not practicable. To produce the picture below I used target-r=40 target-g=50. It depends on the lighting conditions. I have to modify these values permanently.

Best regards,
Wilhelm

https://devtalk.nvidia.com/default/topic/1064908/jetson-projects/jetson-nano-on-a-drone-multicopter/
As you can read in the above thread fawn search and rescue probably will be enhanced with chroma keying. Therefore I’m very interested in getting the function up and running.
Best regards,
Wilhelm

Dear Moderators,
please ask the developers to reprogram the alpha plugin into a hardware alpha (nvalpha). I think that would solve the problem discussed here and further improve the pipeline.

It would also be interesting to see if the problem discussed here is still present on the new Jetson Xavier NX.
Maybe someone can check that.

Thank you in advance,
best regards,
Wilhelm

Not 100% sure, but I’d say:

My pipelines didn’t use any NVIDIA plugin. So the problem seems to be in the alpha plugin from apt package for ubuntu/arm64. This bug may affect any similar ubuntu/arm64 distro and maybe more. Do you have a RPi3 or 4 running an arm64 linux for testing?
Not sure you can expect NVIDIA engineers to fix this for you, although many of them would be able and may at least further comment.

It’s bad that rary didn’t provide more details on his workaround, but he mentionned an interesting fact that alpha got coded as 0xFE on Jetson instead of 0x00 on x86. This might be a coding error in gtsreamer source (bad cast, …) or compiler error…You may investigate, building your own gstreamer of same version or newer, adding debug traces and share on same gstreamer-devel.966125.n4.nabble.com thread.

Unless it comes with a newer gstreamer version where that bug would have been fixed, probably yes it would be the same.

This is an interesting problem. I do think the most important is to report this properly to gstreamer developers for future.

I may help if you wanna try nvivafilter way if you are not impatient. If you want to try this, first have a look to this topic. When you are able to run the sobel filter, you would try to adapt alpha plugin code to oopencv with cuda. Be sure to use RGBA as output format of nvivafilter. You may use cv::cuda for converting from RGBA to RGB, not tried but if RGB2HSV is also supported it may be efficient. Note that you wouldn’t be able to dynamically change settings as nvivafilter plugin source is not fully public, only the custom lib with fixed interface can be rebuilt.
Let us know when you move forward or face a blocking point.