Save 4 usb camera stream to 1 concatenated video

Hi, I’m trying to save 4 different usb camera stream to one video file. I want to concatenate the 4 stream resolution of 1280x720 to one video of 2560x1440.

Can someone know the equivalent this code but for usb camera:

gst-launch-1.0 -e nvcompositor name=mix background-w=2560 background-h=1440
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720
sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720
sink_2::xpos=0 sink_2::ypos=720 sink_2::width=1280 sink_2::height=720
sink_3::xpos=1280 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720
! ‘video/x-raw(memory:NVMM),format=RGBA’ ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=I420’ ! nvv4l2h264enc insert-vui=1 ! h264parse ! matroskamux ! filesink location=test.mkv sync=false
rtspsrc location=rtsp://127.0.0.1:8554/test default-rtsp-version=32 ! application/x-rtp,media=video,encoding-name=H264 ! rtph264depay ! h264parse ! nvv4l2decoder ! ‘video/x-raw(memory:NVMM),format=NV12’ ! queue ! mix.sink_0
rtspsrc location=rtsp://127.0.0.1:8554/test default-rtsp-version=32 ! application/x-rtp,media=video,encoding-name=H264 ! rtph264depay ! h264parse ! nvv4l2decoder ! ‘video/x-raw(memory:NVMM),format=NV12’ ! queue ! mix.sink_1
rtspsrc location=rtsp://127.0.0.1:8554/test default-rtsp-version=32 ! application/x-rtp,media=video,encoding-name=H264 ! rtph264depay ! h264parse ! nvv4l2decoder ! ‘video/x-raw(memory:NVMM),format=NV12’ ! queue ! mix.sink_2
rtspsrc location=rtsp://127.0.0.1:8554/test default-rtsp-version=32 ! application/x-rtp,media=video,encoding-name=H264 ! rtph264depay ! h264parse ! nvv4l2decoder ! ‘video/x-raw(memory:NVMM),format=NV12’ ! queue ! mix.sink_3

Thanks

I only have one USB camera (/dev/video1) so I simulated the 3 other sources with videotestsrc. Note that my USB cam provides YUY2 format that is supported by nvvidconv:

gst-launch-1.0 -ev \
nvcompositor name=mix background-w=2560 background-h=1440 \
 sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720 \
 sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720 \
 sink_2::xpos=0    sink_2::ypos=720 sink_2::width=1280 sink_2::height=720 \
 sink_3::xpos=1280 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720 \
 ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! h264parse ! matroskamux ! filesink location=test.mkv sync=false \
v4l2src device=/dev/video1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_0 \
videotestsrc ! video/x-raw,width=320,height=240 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_1 \
videotestsrc ! video/x-raw,width=320,height=240 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_2 \
videotestsrc ! video/x-raw,width=320,height=240 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_3

Depending on your camera formats, it may need more filters. If you run into issues, post the error messages and output of:

# v4l2-ctl command is provided by apt package v4l-tools
# For /dev/video1:
v4l2-ctl --device  --device=/dev/video1 --list-formats-ext

Hi,
Please try Honey Patouceul’s suggestion. And please share your release version( $ head -1 /etc/nv_tegra_release )

The property background-w and background-h should work only on r32.4.x + this patch:
Gstreamer using nvcomposer to filesink hangs, but nvcomposer to nvoverlaysink works fine - #19 by DaneLLL

Thanks for your reply,
I run this code on 2 camera

gst-launch-1.0 -ev > nvcompositor name=mix background-w=2560 background-h=1440
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720
sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720
! ‘video/x-raw(memory:NVMM),format=RGBA’ ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! h264parse ! matroskamux ! filesink location=test.mkv sync=false
v4l2src device=/dev/video0 ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1’ ! queue ! mix.sink_0
v4l2src device=/dev/video1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1’ ! queue ! mix.sink_1

I get this

Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Caught SIGSEGV
#0 0x0000007f9101cd5c in __waitpid (pid=, stat_loc=0x7fdf041884, options=) at …/sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x0000007f910582a0 in g_on_error_stack_trace ()
#2 0x00000055875a6c3c in ()
#3 0x0000000000008600 in ()
Spinning. Please run ‘gdb gst-launch-1.0 11137’ to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

I run this also as you say :

v4l2-ctl -d1 -D --list-formats-ext

Results

Driver Info (not using libv4l2):
Driver name : uvcvideo
Card type : WDR-200W
Bus info : usb-3610000.xhci-2.3
Driver version: 4.9.201
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘YUYV’
Name : YUYV 4:2:2
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 160x120
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 352x288
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 800x600
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 960x540
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1024x768
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.111s (9.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1280x960
Interval: Discrete 0.143s (7.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.200s (5.000 fps)

Index : 1
Type : Video Capture
Pixel Format: ‘MJPG’ (compressed)
Name : Motion-JPEG
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 160x120
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 352x288
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 800x600
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 960x540
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1024x768
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1280x960
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)

Thanks !
I have this

R32 (release), REVISION: 5.1, GCID: 26202423, BOARD: t186ref, EABI: aarch64, DATE: Fri Feb 19 16:50:29 UTC 2021

But how do you apply this patch once downloaded ?

First try to get one of your cameras to display.

Seems it supports raw mode YUYV 4:2:2 (that is mostly YUY2 in gstreamer) with 640x480@30 fps.

Does this work ?

# If having GUI with X
gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! xvimagesink

# Or
gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! nvvidconv ! nvoverlaysink

Yes the first one works.
I succeed to record one stream with this command :

gst-launch-1.0 v4l2src num-buffers=200 device=/dev/video0 ! ‘video/x-raw, format=YUY2, width=640, height=480, framerate=30/1’ ! videoconvert ! omxh264enc ! qtmux ! filesink location=test.mp4 -ev

So, I’d now suggest you run 4 pipelines just to display for checking the USB bandwidth is ok. It should be ok if your cameras are USB3 and connected through hubs able to do USB3. If not ok, check with lsusb -t.

If you can display the 4 cameras, then you would try:

gst-launch-1.0 -ev \
nvcompositor name=mix \
 sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720 \
 sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720 \
 sink_2::xpos=0    sink_2::ypos=720 sink_2::width=1280 sink_2::height=720 \
 sink_3::xpos=1280 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720 \
 ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! h264parse ! matroskamux ! filesink location=test.mkv sync=false \
v4l2src device=/dev/video0 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_0 \
v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_1 \
v4l2src device=/dev/video2 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_2 \
v4l2src device=/dev/video3 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_3

I try to display 2 cameras in 2 pipelines and it works great,
But when I try your command for 2 cameras it is same the error than at the beginning.

My command :

gst-launch-1.0 -ev \
nvcompositor name=mix \
 sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720 \
 sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720 \
 ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! h264parse ! matroskamux ! filesink location=test.mkv sync=false \
v4l2src device=/dev/video0 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_0 \
v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=1280,height=720,framerate=5/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_1

I changed to 1280*720 resolution and 5FPS.

Hi,
Please check it it works with nvoverlaysink plugin:

gst-launch-1.0 -ev \
nvcompositor name=mix \
 sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720 \
 sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720 \
 ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! nvoverlaysink sync=false \
v4l2src device=/dev/video0 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_0 \
v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=1280,height=720,framerate=5/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1' ! queue ! mix.sink_1

Thanks for your reply.
It runs but my screen goes red like this.

I would like to save to a video file also.

I did what you saied and when I want to run 3 pipeline individually it crashes for the 3rd one.
It returns this :

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
gstv4l2src.c(658): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
EOS on shutdown enabled – waiting for EOS after Error
Waiting for EOS…
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
/GstPipeline:pipeline0/GstQTMux:qtmux0.GstPad:src: caps = video/quicktime, variant=(string)apple
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = video/quicktime, variant=(string)apple

What should I do ?

The red screen with nvoverlaysink is usually caused by a wrong resolution above of what your monitor can do.
Check current resolution with:

xrandr | grep \* | cut -d' ' -f4

and first adapt nvcompositor with videotestsrc inputs to fit your case.

Then replace each videotestsrc with a pipeline providing the expected resolution in NVMM memory into each input and check one by one.

Hi,
There are one Type-A port and two type-C port on Xavier devkit. Are you able to plug in certain cameras to the type-C ports with type-C to type-A adapter? This should avoid issues triggered by insufficient bandwidth.

I have the jetson xavier nx development kit, according to the docs I have 4 USB 3.1.
I connected my 4 USB camera to the jetson.
image

image

How can I know that the usb ports are in usb 3.1 ?

NX has indeed USB3 capability (10000M), but your cameras are running USB2 (480M) only.
You may check details about your camera with:

lsusb -d 1e45:6487 -vvv

I thank you for answering me it’s really nice
I succeed to run on 2 cameras with this command :

gst-launch-1.0 -ev \
v4l2src device=/dev/video0 ! nvvidconv ! "video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1" ! queue ! mix.sink_0 \
v4l2src device=/dev/video1 ! nvvidconv ! "video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,pixel-aspect-ratio=1/1" ! queue ! mix.sink_1 \
nvcompositor name=mix \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720 \
sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 \
! "video/x-raw(memory:NVMM),format=RGBA" ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! h264parse ! matroskamux ! filesink location=test.mkv sync=false 

but when I try on 4 cameras it send error like this:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src3: Failed to allocate required memory.
Additional debug info:
gstv4l2src.c(658): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src3:
Buffer pool activation failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src3: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src3:
streaming stopped, reason not-negotiated (-4)

Have you already encountered this issue ?
Take a hub to improve bandwith ?

It may be a limitation that only 2 USB2 cameras can work with Jetsons.
If you need 4 cameras, you would need to change for USB3 cameras. Not sure if an external hub can help here.

2 Likes

Hi,
Please try Honey Patouceul’s suggestion to use USB3 cameras. And for more than two cameras, it would be better to have externa power supply.

FYR. Here is a similar topic about USB bandwidth:
connected more than two usb cameras problem on deepstream-app (Jetson Nano Dev Kit) - #12 by DaneLLL

1 Like

Thanks for your replies ! It helps me a lot.
I will keep you informed.