Cant merge frames with nvcompositor on jetson nano

[stefan.baar87]

I am trying to merge frames from two usb cameras side by side with gst-launch-1.0 using nvcompositor similar to the example given in the developer wiki

The input looks as as follows:

gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=640 sink_0::height=480 sink_1::xpos=640 sink_1::ypos=0 sink_1::width=640 sink_1::height=480 ! nvoverlaysink v4l2src device=“/dev/video0” io-mode=2 ! ‘image/jpeg,width=640,height=480’ ! nvv4l2decoder mjpeg=1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=RGBA’ ! comp. v4l2src device=“/dev/video1” io-mode=2 ! ‘image/jpeg,width=640,height=480’ ! nvv4l2decoder mjpeg=1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=RGBA’ ! comp.

The error message is as follows:

Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Caught SIGSEGV
#0 0x0000007f9a2add5c in __waitpid (pid=, stat_loc=0x7fda0d2454, options=) at …/sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x0000007f9a2e92e0 in g_on_error_stack_trace ()
#2 0x000000558ccaac3c in ()
#3 0x0000000000008600 in ()
Spinning. Please run ‘gdb gst-launch-1.0 521’ to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

gdb gst-launch-1.0 521 returns the following:

GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “aarch64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from gst-launch-1.0…(no debugging symbols found)…done.
Attaching to program: /usr/bin/gst-launch-1.0, process 521
[New LWP 522]
[New LWP 524]
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/aarch64-linux-gnu/libthread_db.so.1”.
0x0000007f9a2ad5fc in __GI___nanosleep (requested_time=0x7fda0d2468,
remaining=0x7fda0d2478) at …/sysdeps/unix/sysv/linux/nanosleep.c:28
28 …/sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.

I am able to run both cameras simultaneously in two separate windows with:

gst-launch-1.0 v4l2src device=“/dev/video0” ! ‘image/jpeg,width=1920,height=1080’ ! nvv4l2decoder mjpeg=1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=RGBA’ ! nvoverlaysink

I wonder why, I cant merge the video streams…

[DaneLLL]

Hi,
Looks like you set USB cameras to generate MJPEG stream. Generally USB cameras can also generate YUV422 such as YUYV or UYVY. Please get supported formats by executing $ v4l2-ctl --list-formats-ext. If it supported YUV422, we suggest try the format for comparison. To check if the failure is specific to MJPEG stream.

And please share your release version:

$ cat /etc/nv_tegra_release

[stefan.baar87]

Thank you for the quick reply.
I think my USB cameras can only generate MJPEG streams …but I am not sure.

v4l2-ctl --list-formats-ext returns the following:

ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘MJPG’ (compressed)
Name : Motion-JPEG
Size: Discrete 4032x3040
Interval: Discrete 0.100s (10.000 fps)
Size: Discrete 3840x2160
Interval: Discrete 0.050s (20.000 fps)
Size: Discrete 2592x1944
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 2560x1440
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)
Size: Discrete 1600x1200
Interval: Discrete 0.020s (50.000 fps)
Size: Discrete 1280x960
Interval: Discrete 0.010s (100.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.010s (100.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.013s (80.000 fps)

The release version is:

R32 (release), REVISION: 5.2, GCID: 27767740, BOARD: t210ref, EABI: aarch64, DATE: Fri Jul 9 16:01:52 UTC 2021

ffmpeg -f video4linux2 -list_formats all -i /dev/video1 returns the following:

ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libavresample 3. 7. 0 / 3. 7. 0
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
[video4linux2,v4l2 @ 0x55a1452600] Compressed: mjpeg : Motion-JPEG : 4032x3040 3840x2160 2592x1944 2560x1440 1920x1080 1600x1200 1280x960 1280x720 640x480

[stefan.baar87]

I have prepared another Jetson Nano, but instead of using jetpack 4.5, I installed the newest jetpack (4.6) and luckily here everything works.
However, merging three camera streams (720p) with nvcompositor and saving to file with nvjpegenc seems to drop a few frames

jetpack 4.5: streamer 1.14.5 is for some reason not working …

[DaneLLL]

Hi,
Good to know it works on latest Jetpack 4.6. For performance issue, you may try to run hardware converter(VIC engine) at max clock for a try. Please refer to steps in
Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5? - #3 by DaneLLL