Hardware Accelerated GStreamer Plugins

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I am close to the end of my project, but one thing still is stopping me. I have yet to find the most up to date plugin that can achieve the following: Combine multiple video streams (from nvarguscamerasrc) and combine them into a format that can then be passed into nvv4l2h265enc, and then be saved to file.

I thought glvideomixer would be good enough for this purpose, but running the following command gives several errors:

gst-launch-1.0 -e glvideomixer name=mix sink_0::ypos=0 sink_1::ypos=1024 ! qtmux ! filesink location=filename.mp4 nvarguscamerasrc sensor-mode=0 sensor-id=0 ! "video/x-raw(memory:NVMM), width=(int)5328, height=(int)4608, format=(string)NV12, framerate=(fraction)15/1" ! nvvidconv ! "video/x-raw, format=NV12, width=1024, height=1024" ! mix.sink_0 nvarguscamerasrc sensor-mode=0 sensor-id=1 ! "video/x-raw(memory:NVMM), width=(int)5328, height=(int)4608, format=(string)NV12, framerate=(fraction)15/1" ! nvvidconv ! "video/x-raw, format=NV12, width=(int)1024, height=(int)1024" ! mix.sink_1

Errors can be seen in these two screenshots:
Imgur
Imgur

Any ideas?

2

Probably the plugin you’re looking for is nvcompositor:

gst-inspect-1.0 nvcompositor

searching this forum you may find various examples.

3

I found one single example that worked in the context I was looking for.

I have another question though, as I am unsure if what I found had the correct answer or not: Is there a maximum resolution to the hardware accelerated nvv4l2h265enc and nvv4l2h264enc encoders? If so, what is it?

4

You would try something like:

gst-launch-1.0 -e \
nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420, width=3840, height=1080' ! nvv4l2h265enc ! h265parse ! matroskamux ! filesink location=test_h265.mkv    \
nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_0   \
nvarguscamerasrc sensor_id=1 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_1

I think that nvv4l2h264enc has a maximum width of 4096, and for nvv4l2h265enc it should be 8192.
Never hit the limit in height, I cannot say but it’s easy to try.

5

I saw this post right here Using h265 with resolutions bigger than 4096 - #10 by DaneLLL, and I am not entirely sure which one is the correct maximum value, but I haven’t been able to use more than the values specified on that forum. I will get the command I have been using so far in a second.

6

I am also making sure that you know that I am using a TX2 on JetPack 4.5.1, but that shouldn’t change anything.

gst-launch-1.0 nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=0 sink_1::ypos=2600 ! nvvidconv ! nvv4l2h265enc ! h265parse ! qtmux ! filesink location=stereo.mp4 
nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=2600,height=2600,framerate=1/1' ! nvvidconv ! nvcomp.sink_0 
nvarguscamerasrc sensor-id=1 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1496,height=1496,framerate=1/1' ! nvvidconv ! nvcomp.sink_1

Increasing the height of either sources even by 1 causes it to fail entirely.

Your command also does not work, if I set the width to be 3840 on both sources, even though that would not exceed the 8192 of h265encoding.

7

Sorry, I’ve only quickly tested with AGX Xavier running R32.6.1. Not sure if the NVENC HW is the same in TX2 and Xavier SoCs. The available memory may also be different.

Note that odd widh may not be supported. Better try increasing by steps of 2.

Just sharing what works on my AGX Xavier capturing 1080p30, composing and rescaling to 7160x4320, encoding into H265 and storing into mkv container file:

gst-launch-1.0 -e \
nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920\
 ! 'video/x-raw(memory:NVMM), width=3840,height=1080' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=NV12, width=7160, height=2160, pixel-aspect-ratio=1/1' ! nvv4l2h265enc ! h265parse ! matroskamux ! filesink location=test_h265.mkv    \
nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_0   \
nvarguscamerasrc sensor_id=1 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_1

and play with:

gst-launch-1.0 filesrc location=test_h265.mkv ! matroskademux ! h265parse ! nvv4l2decoder ! nvegltransform ! nveglglessink

This might be the cause. Can you run at least 10 fps ?

8

Hi,
8K video encoding is not supported on TX2. It is a constraint in hardware. Please keep width <= 4096.

9

Maximum height, after my testing ended up being 8192, following this command.

gst-launch-1.0 nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=0 sink_1::ypos=4096 ! nvvidconv ! nvv4l2h265enc ! h265parse ! qtmux ! filesink location=stereo.mp4 \
nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=4096,height=4096,framerate=1/1' ! nvvidconv ! nvcomp.sink_0 \
nvarguscamerasrc sensor-id=1 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=4096,height=4096,framerate=1/1' ! nvvidconv ! nvcomp.sink_1

For our project, we need at least 1 fps for recording, so I am using this as a baseline. Eventually, if possible, we might tick it up to 2, maybe 3 fps.

I think I was just doing something wrong, although not sure besides increasing height and width by only 1, when that is not supported.