Gstreamer get 2 4k usb cameras simultaneously and put them side by side

aandymsg · March 4, 2022, 2:52pm

Hello everybody

This is my first post here so please bear with me.
So i have tested my jetson nano with 2 logitech c270 usb cameras attached and managed to get video from both cameras at once and put them side by side to create a single video using this command:

sudo gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,format=YUY2,width=800,height=600,framerate=20/1' ! videobox left=-800 border-alpha=0 ! queue ! videomixer name=mix ! videoconvert ! xvimagesink sync=false v4l2src device=/dev/video1 ! 'video/x-raw,format=YUY2,width=800,height=600,framerate=20/1' ! videobox left=0 ! queue ! mix.

However, I am now trying to do the same with some higher quality cameras. I have 2 usb 4K cameras capable of YUYV4.2.2 and MJPG, and i try to do the same as i did with the logitech ones: However, trying the same command spits out an error that videobox0 can’t handle caps. I’ve been trying all kinds of variations but all to no success. Anyone has any ideea how could i achieve the same but in higher resolution? I have managed at some point to get the video but only in YUYV which caps my fps at 1 for 4K resolution which is pretty bad.

Basically what i need is to get the 4k video in MJPG format from both cameras simultaneously with gstreamer and put the videos side by side to create a single video in the stereoscopis format. Can anyone give a hand!? Thaanks a bunch.

Regards
Andy

DaneLLL · March 7, 2022, 5:22am

Hi,
You can try the command and check if the two cameras can achieve target fps:

gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=3840,height=2160,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 v4l2src device=/dev/video1 io-mode=2 ! image/jpeg,width=3840,height=2160,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

One 4Kp30 should be achievable on Jetson Nano. Two 4Kp30 may hit hardware constraint.

aandymsg · March 8, 2022, 1:09pm

It worked. Both cameras started and the output of the command was:

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1/GstFakeSink:fakesink1: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 19, dropped: 0, current: 35,97, average: 35,97
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 19, dropped: 0, current: 37,65, average: 37,65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 34, dropped: 0, current: 29,94, average: 33,81
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 35, dropped: 0, current: 29,84, average: 32,88
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 50, dropped: 0, current: 30,10, average: 32,53
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 51, dropped: 0, current: 30,09, average: 31,95
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 66, dropped: 0, current: 30,08, average: 31,90
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 67, dropped: 0, current: 30,08, average: 31,48
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 82, dropped: 0, current: 29,77, average: 31,46
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 83, dropped: 0, current: 29,77, average: 31,14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 98, dropped: 0, current: 30,19, average: 31,24
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 99, dropped: 0, current: 30,18, average: 30,98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 113, dropped: 0, current: 29,93, average: 31,06
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 114, dropped: 0, current: 29,94, average: 30,84
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 129, dropped: 0, current: 30,10, average: 30,94
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 130, dropped: 0, current: 30,11, average: 30,75
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 145, dropped: 0, current: 30,11, average: 30,85
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 146, dropped: 0, current: 30,10, average: 30,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 161, dropped: 0, current: 29,86, average: 30,75
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 162, dropped: 0, current: 29,86, average: 30,59
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 177, dropped: 0, current: 30,08, average: 30,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 178, dropped: 0, current: 30,07, average: 30,55
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 192, dropped: 0, current: 29,98, average: 30,63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 193, dropped: 0, current: 29,98, average: 30,50
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 208, dropped: 0, current: 30,08, average: 30,59

I stopped it manually :

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 843, dropped: 0, current: 29,20, average: 30,11
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 840, dropped: 0, current: 30,10, average: 29,91
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:28.713031913
Setting pipeline to NULL ...
Freeing pipeline ...

I use a Jetson 4 GB. I also have a 2GB one but i think on the 2G one i won’t be able to do this.
Attached image with cameras blue control led lighs that shows they are working and are accessed.

aandymsg · March 8, 2022, 1:22pm

I have tested several times. Occasionally when starting the command i get this shortly after starting:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could not read from resource.
Additional debug info:
gstv4l2bufferpool.c(1127): gst_v4l2_buffer_pool_poll (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
poll error 1: Success (0)
Execution ended after 0:00:04.291865421
Setting pipeline to NULL ...
Freeing pipeline ...
jetson1@jetson1-desktop:~$

So i imagine i may hit those hardware constraints you mentioned.

DaneLLL · March 9, 2022, 1:29am

Hi,
For putting the two sources side by side, please try nvcompositor. You can refer to this topic:
Gst for merging 4 webcams sometimes works, and sometimes it doesn't

And need the patch to rebuild nvcompositor:
Gst for merging 4 webcams sometimes works, and sometimes it doesn't - #5 by DaneLLL

ances.ai · March 23, 2022, 7:53pm

Hello! I have to say that there are an error with Gst for merging 4 webcams sometimes works, and sometimes it doesn't

The patch doesn’t work. There are no errors, but the filesink file is not filled.

I managed to get it to work with a change in the code:

gst-launch-1.0 nvcompositor background-w=2560  background-h=1440 name=comp sink_0::xpos=0  sink_0::ypos=0    sink_0::width=1280 sink_0::height=720 \
                       sink_1::xpos=1280 sink_1::ypos=0    sink_1::width=1280 sink_1::height=720 \
                       sink_2::xpos=0    sink_2::ypos=720 sink_2::width=1280 sink_2::height=720 \
                       sink_3::xpos=1280 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720 \
! nvvidconv ! 'video/x-raw(memory:NVMM), width=2560, height=1440, framerate=60/1, format=RGBA' \
! nvvidconv ! omxh264enc control-rate=2 bitrate=20000000 ! queue ! mux. alsasrc device='hw:2,0' ! audio/x-raw,width=16,depth=16,rate=32000,channels=2 ! queue ! audioconvert ! audioresample ! voaacenc ! aacparse ! qtmux name=mux ! filesink location=/home/ances/0.mp4  -e \
v4l2src device=/dev/video0 ! image/jpeg,format=MJPG,width=1280,height=720,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_0 \
v4l2src device=/dev/video1 ! image/jpeg,format=MJPG,width=1280,height=720,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_1 \
v4l2src device=/dev/video2 ! image/jpeg,format=MJPG,width=1280,height=720,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_2 \
v4l2src device=/dev/video3 ! image/jpeg,format=MJPG,width=1280,height=720,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_3

The part that says that is important:

'video/x-raw(memory:NVMM), width=2560, height=1440, framerate=60/1, format=RGBA'

The intermittent bug happend if you use this:

'...format=NV12'

I wish I could correct it in the original post.

system · April 6, 2022, 7:53pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.