Nvcompositor - positioning frames failure

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi
Struggling with frame positioning with Nvcompositor element. Wanting to combine different resolutions for different frames, and result should end in a 21:9 ratio frame for displaying on a 2560x1080 monitor.

I’ve achieved to fill a whole screen with 6 video sources, with the use of this pipeline:

gst-launch-1.0 nvcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1080 sink_0::height=1920 sink_0::zorder=1 \
sink_1::xpos=1080 sink_1::ypos=0 sink_1::width=1080 sink_1::height=1920 sink_1::zorder=1 \
sink_2::xpos=2160 sink_2::ypos=0 sink_2::width=1280 sink_2::height=720 sink_2::zorder=1 \
sink_3::xpos=3440 sink_3::ypos=0 sink_3::width=1280 sink_3::height=720 sink_3::zorder=1 \
sink_4::xpos=2160 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720 sink_4::zorder=2 \
sink_5::xpos=3440 sink_5::ypos=720 sink_5::width=1280 sink_5::height=720 sink_5::zorder=2 \
! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420, width=(int)2560, height=(int)1080" ! nvv4l2h264enc ! 'video/x-h264' ! h264parse ! qtmux ! filesink location=placement_current.mp4 -e \
videotestsrc pattern=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=720, height=1280" ! comp. \
videotestsrc pattern=0 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=720, height=1280" ! comp. \
videotestsrc pattern=3 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp. \
videotestsrc pattern=4 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp. \
videotestsrc pattern=5 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp. \
videotestsrc pattern=6 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp.

This pipeline results in this frame:

current_placement
current_placement931×397 137 KB

However, I now want to reposition the pictures to fit to smaller ones on the left side (over/under) followed by the two vertical pictures in the middle, and again two smaller ones (as red/blue frames position). Trying to just change the position values makes the resulting frame repositioning it self, making an annoying black, empty border on the right side, with the same amount of assumed pixels (width) as one of the vertical frames.
By running this pipeline, it results in the following picture:

gst-launch-1.0 nvcompositor name=comp \
sink_0::xpos=1280 sink_0::ypos=0 sink_0::width=1080 sink_0::height=1920 sink_0::zorder=2 \
sink_1::xpos=2360 sink_1::ypos=0 sink_1::width=1080 sink_1::height=1920 sink_1::zorder=2 \
sink_2::xpos=0 sink_2::ypos=0 sink_2::width=1280 sink_2::height=720 sink_2::zorder=1 \
sink_3::xpos=3440 sink_3::ypos=0 sink_3::width=1280 sink_3::height=720 sink_3::zorder=1 \
sink_4::xpos=0 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720 sink_4::zorder=1 \
sink_5::xpos=3440 sink_5::ypos=720 sink_5::width=1280 sink_5::height=720 sink_5::zorder=1 \
! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420, width=(int)2560, height=(int)1080" ! nvv4l2h264enc ! 'video/x-h264' ! h264parse ! qtmux ! filesink location=placement_new.mp4 -e \
videotestsrc pattern=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=720, height=1280" ! comp. \
videotestsrc pattern=0 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=720, height=1280" ! comp. \
videotestsrc pattern=3 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp. \
videotestsrc pattern=4 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp. \
videotestsrc pattern=5 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp. \
videotestsrc pattern=6 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=960, height=540" ! comp.

new_placement
new_placement1307×438 206 KB

The aspect ratio also seems to increase, making the total frame wider, which is not what I want. Any suggestions to make it fit (almost) as perfect as in the first picture? For me, logically the two pipelines are just the same, except a few minor changes in the frames placement. The only changes are x_pos and y_pos of all of the sinks.

3

Hi,
The position of each video plane cannot be changed dynamically. This use-case is not supported. For changing the layout, please re-initialize the gstreamer pipeline.

4

Main issue may be that you don’t specify output resolution of nvcompositor (4720x1920). The caps are thus taken as same size of your rescaled size (2560x1080).

You may try:

# Current placement:
gst-launch-1.0 \
nvcompositor name=comp \
 sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1080 sink_0::height=1920 sink_0::zorder=1 \
 sink_1::xpos=1080 sink_1::ypos=0   sink_1::width=1080 sink_1::height=1920 sink_1::zorder=1 \
 sink_2::xpos=2160 sink_2::ypos=0   sink_2::width=1280 sink_2::height=720  sink_2::zorder=1 \
 sink_3::xpos=3440 sink_3::ypos=0   sink_3::width=1280 sink_3::height=720  sink_3::zorder=1 \
 sink_4::xpos=2160 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720  sink_4::zorder=2 \
 sink_5::xpos=3440 sink_5::ypos=720 sink_5::width=1280 sink_5::height=720  sink_5::zorder=2 \
 ! "video/x-raw(memory:NVMM), format=RGBA, width=4720, height=1920" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420, width=(int)2560, height=(int)1080, pixel-aspect-ratio=1/1" ! nvv4l2h264enc ! h264parse ! qtmux ! queue ! filesink location=placement_current.mp4 -e \
videotestsrc pattern=1 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=1080, height=1920" ! queue ! comp.sink_0 \
videotestsrc pattern=0 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=1080, height=1920" ! queue ! comp.sink_1 \
videotestsrc pattern=3 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_2 \
videotestsrc pattern=4 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_3 \
videotestsrc pattern=5 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_4 \
videotestsrc pattern=6 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_5

# New placement
gst-launch-1.0 \
nvcompositor name=comp \
 sink_0::xpos=1280 sink_0::ypos=0   sink_0::width=1080 sink_0::height=1920 sink_0::zorder=2 \
 sink_1::xpos=2360 sink_1::ypos=0   sink_1::width=1080 sink_1::height=1920 sink_1::zorder=2 \
 sink_2::xpos=0    sink_2::ypos=0   sink_2::width=1280 sink_2::height=720  sink_2::zorder=1 \
 sink_3::xpos=3440 sink_3::ypos=0   sink_3::width=1280 sink_3::height=720  sink_3::zorder=1 \
 sink_4::xpos=0    sink_4::ypos=720 sink_4::width=1280 sink_4::height=720  sink_4::zorder=1 \
 sink_5::xpos=3440 sink_5::ypos=720 sink_5::width=1280 sink_5::height=720  sink_5::zorder=1 \
 ! "video/x-raw(memory:NVMM), format=RGBA, width=4720, height=1920" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420, width=(int)2560, height=(int)1080, pixel-aspect-ratio=1/1" ! nvv4l2h264enc ! h264parse ! qtmux ! queue ! filesink location=placement_new.mp4 -e \
videotestsrc pattern=1 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=1080, height=1920" ! queue ! comp.sink_0 \
videotestsrc pattern=0 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv flip-method=3 ! "video/x-raw(memory:NVMM), format=I420, width=1080, height=1920" ! queue ! comp.sink_1 \
videotestsrc pattern=3 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_2 \
videotestsrc pattern=4 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_3 \
videotestsrc pattern=5 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_4 \
videotestsrc pattern=6 is-live=1 ! 'video/x-raw, width=1920, height=1080' ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420, width=1280, height=720" ! queue ! comp.sink_5

Also note that H264 encoding quality may be low with default parameters. If this is a concern for this case, you may also try increasing encoder bitrate and/or profile such as:

# 32 Mb with H264 High Profile 
... ! nvv4l2h264enc bitrate=32000000 profile=4 ! ...
5

Hi
Thanks both for answering. We managed to find out where it failed. By removing the rescaling of width and height of all camera sources just before each comp (width=1080, height=1920" ! queue ! comp.sink_0) it worked out as wanted.
Adding caps to the nvcompositor (width=4720, heigth=1920) does not work, and the pipeline automatically ends it self. But it does now work going directly to nvvidconv and then rescaling to 2560x1080.

I’m aware of the H264 encoding parameters, but it was taken away for file writing purposes since it does give good enough quality. However thanks.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.