NVcompositor ignores buffer timestamps

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi,

We’re compositing 4 synchronised v4l2 camera sources in a Gstreamer pipeline. To improve the performance we substituted the Gstreamer ‘compositor’ element for the NVIDIA hardware accelerated ‘nvcompositor’ version.

It seems that nvcompositor doesn’t synchronise inputs from the buffer timestamps. I’ve included a screenshot showing the issue. The buffer timestamps are drawn with the gstreamer element ‘timeoverlay’

composited screengrab
composited screengrab1920×1080 113 KB

Is this a known limitation of nvcompositor or is there something I’m missing? This is running on a Jetson AGX Orin.

Thanks
Russell

2

Hi @PeartreeStudios,

From what I understand, nvcompositor does not have synchronization capabilities, which means that you will probably need your streams to already be synchronized before composing them.

When you say:

Does this mean your cameras are already synchronized to each other?

Is it possible for you to test using nvarguscamerasrc, perhaps something like this?

gst-launch-1.0  \
nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! comp.sink_0 \
nvarguscamerasrc sensor-id=1 ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! comp.sink_1 \
nvarguscamerasrc sensor-id=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! comp.sink_2 \
nvarguscamerasrc sensor-id=3 ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! comp.sink_3 \
nvcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 \
sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 \
sink_2::xpos=0 sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080 \
sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 \
! "video/x-raw(memory:NVMM), width=3840, height=2160" ! nvoverlaysink

Regards,
Francis Guindon

Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: http://www.ridgerun.com/

4

Hi,
Synchronization is done in gstreamer frameworks. If frames from each data source are ready, it calls gst_nvcompositor_aggregate_frames().

The source code of nvcompositor is public. For further checking, you may download the source code, add debug prints, and rebuild the plugin.

All public source is in

Jetson Linux | NVIDIA Developer
Driver Package (BSP) Sources

5

We’re using a 4x GMSL camera kit for the AGX Orin which uses a single clock to drive the CSI sensor modules. I believe they are capturing synchronised because when I look at at camera frames with the same buffer timestamp, the image captured matches. Having said that, I’m not sure timeoverlay uses the v4l2 captured frame timestamp, but rather gstreamers own buffer timestamp.

The Gstreamer pipeline is:

gst-launch-1.0 -e \
  nvcompositor name=comp \
    sink_0::xpos=0     sink_0::ypos=0     sink_0::width=1920 sink_0::height=1080 \
    sink_1::xpos=1920  sink_1::ypos=0     sink_1::width=1920 sink_1::height=1080 \
    sink_2::xpos=0     sink_2::ypos=1080  sink_2::width=1920 sink_2::height=1080 \
    sink_3::xpos=1920  sink_3::ypos=1080  sink_3::width=1920 sink_3::height=1080 \
  ! nvvidconv \
  ! queue \
  ! nvv4l2h264enc \
  ! h264parse \
  ! qtmux \
  ! filesink location=./output.mp4 sync=false \
  v4l2src device=/dev/video0 do-timestamp=true ! videorate ! capsfilter caps="video/x-raw,format=\(string\)UYVY,width=1920,height=1080,framerate=30/1" ! timeoverlay \
    ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! identity sync=true ! comp.sink_0 \
  v4l2src device=/dev/video1 do-timestamp=true ! videorate ! capsfilter caps="video/x-raw,format=\(string\)UYVY,width=1920,height=1080,framerate=30/1" ! timeoverlay \
    ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! identity sync=true ! comp.sink_1 \
  v4l2src device=/dev/video2 do-timestamp=true ! videorate ! capsfilter caps="video/x-raw,format=\(string\)UYVY,width=1920,height=1080,framerate=30/1" ! timeoverlay \
    ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! identity sync=true ! comp.sink_2 \
  v4l2src device=/dev/video3 do-timestamp=true ! videorate ! capsfilter caps="video/x-raw,format=\(string\)UYVY,width=1920,height=1080,framerate=30/1" ! timeoverlay \
    ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! identity sync=true ! comp.sink_3

In case you’re wondering why we have videorate in there, it’s because the underlying camera’s only expose 1080p60, so we’re using videorate to cut it down to 30fps.

Thanks

6

Hi, when you say ‘if frames from each data source are ready…’ do you mean it waits for frames with matching buffer timestamps?

I’ll have a look at the nvcompositor source, thanks. Is there a mirror on GitHub by any chance?

Thanks

7

Hi,
We know there is synchronization in gstreamer frameworks but uncertain how the sources are handled. You may need to check source code of gstreamer:

Building from source using Meson

We don’t have github to nvcompositor plugin. Please download the source code package. You may print out pts and dts of each GstBuffer for check.