Accelerated chroma keying for Jetson Xavier NX with gstreamer

Robotics & Edge Computing Jetson Systems Jetson Xavier NX
1

Hi everyone,

I am currently trying to implement accelerated chroma keying on the jetson xavier nx in order to remove some blue pixel on a desktop stream and replace them with a PAL video stream. I use the v4l2src plugin form gstreamer to stream the videos.
I tried to use the alpha plugin from gstreamer. It worked but I have less than one fps.

I am thinking about using nvivafilter but I don’t know how to get 2 inputs with this plugin.

I have already seen all the previous discussions on this forum but I can’t really distinguish any lasting solution in those.

If you have any other solution, I m all ears.

Thanks in advance

2

Hello @hugo.delaruelle,

How is everything going?

Can you please share your current pipeline?

We want to first fully understand your situation before trying to propose any solutions.

best regards,
Andrew
Embedded Software Engineer at ProventusNova

3

Hi, here is my pipeline using the alpha plugin.

gst-launch-1.0 v4l2src device=/dev/video2 ! mixer.sink_0 v4l2src device=/dev/video0 ! alpha method=custom target-b=180 target-r=10 target-g=10 ! mixer.sink_1 videomixer name=mixer sink_0::zorder=0 sink_1::zorder=1 ! nv3dsink

But the alpha plugins’s performance is poor since it doesn’t seem to be GPU accelerated. I am looking for a more efficient solution to perform chroma keying.

4

Hi @hugo.delaruelle,

Thanks for sharing the pipe.

Would you be opposed to giving this pipe a try?

gst-launch-1.0 v4l2src device=/dev/video2 ! queue ! mixer.sink_0 v4l2src device=/dev/video0 ! queue ! alpha method=custom target-b=180 target-r=10 target-g=10 ! queue ! mixer.sink_1 videomixer name=mixer sink_0::zorder=0 sink_1::zorder=1 ! nv3dsink

The queues might help with frame-rate, just be aware that you might see a bit more latency.

That being said, you are correct.
By having a pipe optimized for the NVIDIA HW you will get the best performance.
That would require a bit more work though.

If you need help with that, please do not hesitate to reach out, we would love to give you a hand. There are a couple of options we can try:

  1. Modifying the nvivafilter to support more inputs.
  2. Creating a custom GStreamer element with HW optimized chroma keying.

Hit us up if you want to get that pipe optimized ASAP.

best regards,
Andrew
Embedded Software Engineer at ProventusNova

6

Hi,
For the use-case, there is no existing implementation with hardware acceleration. A possible solution is to implement it through jetson_multimedia_api like:

  1. Capture frame data to NvBufSurface. May refer to

/usr/src/jetson_multimedia_api/samples/12_v4l2_camera_cuda

  1. Access the buffer on GPU through NvBufSurface APIs
  2. Implement CUDA code for the use-case
7

Hi,
Thank you for your answer.
I am trying to display the stream of a PAL camera with this sample but it seems that the video is not deinterlaced.
Is it possible to perform deinterlacing ? If yes, what modifications do I need to operate in the code ?

8

Hi,
There is no hardware acceleration for deinterlacing in the use-case. You would need to use software solution. So your solution looks to be optimal on Xavier NX. One more thing to try is to run $ sudo jetson_clocks to fix CPU cores at maximum clock.

9

My video grabber driver is not compatible with DMA BUF buffers. I changed for MMAP buffers and now everything is working properly.