GStreamer: Multiple webcam sources, Picture in Picture to mux on a Jetson Nano, then to be used as a pipeline with belabox

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

GStreamer: Multiple webcam sources, Picture in Picture to mux on a Jetson Nano, then to be used as a pipeline with belabox >> belabox.net

I’m currently trying to pull two different usb webcams into a pipeline and create a Picture-in-Picture composite, but I keep getting an error like this:

GStreamer Error: gstreamer error from v4l2src1

This is the current pipeline I’m working on that doesn’t work;

v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvvidconv ! queue ! comp.sink_0 !
v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! queue ! comp.sink_1 !

nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' !
queue ! identity name=v_delay signal-handoffs=TRUE !

nvv4l2decoder mjpeg=1 enable-max-performance=true ! nvvidconv !
textoverlay text='' valignment=top halignment=right font-desc="Monospace, 5" name=overlay ! queue !

nvvidconv interpolation-method=5 !
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps !
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux.

alsasrc device=hw:2 ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 !
audioconvert ! opusenc bitrate=320000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux.

mpegtsmux name=mux !
appsink name=appsink

Please note; I have two separate pipelines that can use each webcam, so I know the cameras work and can be used, I just need them to be in the same pipeline have a picture-in-picture composite and work in the belabox environment.

For reference this is a working usb webcam pipeline that works with belabox:

v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 !
identity name=v_delay signal-handoffs=TRUE !
nvv4l2decoder mjpeg=1 enable-max-performance=true ! nvvidconv !
textoverlay text='' valignment=top halignment=right font-desc="Monospace, 5" name=overlay ! queue !
nvvidconv interpolation-method=5 !
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps !
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux.
alsasrc device=hw:2 ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 !
audioconvert ! voaacenc bitrate=128000 ! aacparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux.
mpegtsmux name=mux !
appsink name=appsink

Any Ideas?

2

If you look at SINK capabilities of nvcompositor:

gst-inspect-1.0 nvcompositor

you would see it only supports input from NVMM memory with formats RGBA, I420 or NV12.
If reading from a jpeg encoding camera, you would need to decode into raw video. Probably best option would be trying nvv4l2decoder:

v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_0

Note that this sub-pipeline is complete as having source and sink, so it may fail if using a ‘!’ just after.

Also, same for videobox. You would need to decode into raw video. Using nvv4l2decoder or nvjpegdec would output into NVMM memory but videobox only supports system memory, so you would have to copy NVMM->system before and system->NVMM after with nvvidconv. This may not be that efficient (or try jpegdec as decoder, it would output to system memory so you would save NVMM->system copy but that may use CPU in turn).
nvvidconv can be used to crop (although right and bottom have different meaning as being source image coordinates rather than margins), but I think that expanding has been limited a few L4T releases ago, so it might not be an alternative for adding borders.

3

Thank you for this, I was just going off of other examples, to be honest I don’t really know how to structure a pipeline, or what a “!” does, so what would be a full example of this so I can test out and see if it works?

Kind Regards,
-b3ck

4

I tried running it with your suggestion, here is the log:

Sep 08 06:48:43 belabox belaUI[3824]: Gstreamer pipeline: v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_0
Sep 08 06:48:43 belabox belaUI[3824]: v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_1
Sep 08 06:48:43 belabox belaUI[3824]: nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 sink_1::zorder=2 ! ‘video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1’ ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=NV12’ !
Sep 08 06:48:43 belabox belaUI[3824]: queue ! identity name=v_delay signal-handoffs=TRUE !
Sep 08 06:48:43 belabox belaUI[3824]: nvv4l2decoder mjpeg=1 enable-max-performance=true ! nvvidconv !
Sep 08 06:48:43 belabox belaUI[3824]: queue !
Sep 08 06:48:43 belabox belaUI[3824]: nvvidconv interpolation-method=5 !
Sep 08 06:48:43 belabox belaUI[3824]: nvv4l2h265enc control-rate=1 qp-range=“28,50:0,38:0,50” iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps !
Sep 08 06:48:43 belabox belaUI[3824]: h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux.
Sep 08 06:48:43 belabox belaUI[3824]: alsasrc device=hw:2 ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 !
Sep 08 06:48:43 belabox belaUI[3824]: audioconvert ! opusenc bitrate=320000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux.
Sep 08 06:48:43 belabox belaUI[3824]: mpegtsmux name=mux !
Sep 08 06:48:43 belabox belaUI[3824]: appsink name=appsink
Sep 08 06:48:43 belabox belaUI[3824]: (belacoder:27340): GStreamer-CRITICAL **: 06:48:43.981: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed
Sep 08 06:48:43 belabox belaUI[3824]: (belacoder:27340): GStreamer-CRITICAL **: 06:48:43.981: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed

5

It is used to separate plugins in a pipeline. It may further be used for setting caps between plugins.

First, check that you can read images from both camera and sound from microphone. Assuming you locally connected to Nano with X running and sound working, you would try (use Ctrl-C for stopping):

# Cam1
gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink

# Cam2
gst-launch-1.0 -v v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink

# Audio
gst-launch-1.0 -v alsasrc device=hw:2 ! autoaudiosink

If all 3 work, you would first try to compose both videos, adding textoverlay on both and videobox on cam2. It has been succesfully tested on AGX Orin, not sure it works on Nano. So you would first boost Nano:

# Switch to MAXN mode
sudo nvpmodel -m0

# Boost clocks
sudo jeston_clocks

and measure the achieved framerate with fpsdisplaysink:

gst-launch -v \
   v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! textoverlay text=\"cam1\" valignment=top halignment=right font-desc="Monospace, 5" name=overlay1 ! queue ! nvvidconv interpolation-method=5 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! queue ! comp.sink_0  \
   v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! videobox left=-4 right=-4 top=-4 bottom=-4 ! textoverlay text="cam2" valignment=top halignment=right font-desc="Monospace, 10" name=overlay2 ! queue ! nvvidconv interpolation-method=5 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! queue ! comp.sink_1 \
   nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! fpsdisplaysink text-overlay=0 video-sink=xvimagesink

If not working, you may try to decrease framerate to 30/1 (or less) if both your cameras support this.

Assuming it is working, you would encode video as H265 and add sound and mux both into mpegts. In order to check the stream, we will locally stream it using RTP/MP2T so that you’ll be able to check:

gst-launch -v \
  v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! textoverlay text=\"cam1\" valignment=top halignment=right font-desc="Monospace, 5" name=overlay1 ! queue ! nvvidconv interpolation-method=5 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! queue ! comp.sink_0  \
  v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! videobox left=-4 right=-4 top=-4 bottom=-4 ! textoverlay text="cam2" valignment=top halignment=right font-desc="Monospace, 10" name=overlay2 ! queue ! nvvidconv interpolation-method=5 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! queue ! comp.sink_1 \
  nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. \
  alsasrc ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! audioconvert ! opusenc bitrate=320000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux. \
  mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5004

You would receive with:

gst-launch-1.0 -v udpsrc port=5004 ! application/x-rtp,encoding-name=MP2T,payload=33,clock-rate=90000 ! rtpjitterbuffer latency=200 ! rtpmp2tdepay ! tsparse ! tsdemux name=demux ! queue ! audio/x-opus ! decodebin ! audioconvert ! audioresample ! autoaudiosink  demux. ! queue ! video/x-h265 ! h265parse ! nvv4l2decoder ! nvvidconv ! xvimagesink

# or 
ffmplay rtp://127.0.0.1:5004

# vlc may also work but may have worse performance
cvlc rtp://127.0.0.1:5004

If that works, you have most of the pipeline. What you may have to do is adapting the syntax from gst-launch to code pipeline. You would remove single quotes (these were used for preventing shell from interpreting the parenthesis), remove ‘\’ , and add a ‘\’ before double quotes inside the pipeline, and replace with appsink at the tail:

pipeline_str = "v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! textoverlay text=\"cam1\" valignment=top halignment=right font-desc=\"Monospace, 5\" name=overlay1 ! queue ! nvvidconv interpolation-method=5 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1 ! queue ! comp.sink_0        v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! videobox left=-4 right=-4 top=-4 bottom=-4 ! textoverlay text=\"cam2\" valignment=top halignment=right font-desc=\"Monospace, 10\" name=overlay2 ! queue ! nvvidconv interpolation-method=5 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1 ! queue ! comp.sink_1         nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 sink_1::zorder=2 ! video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1 ! nvvidconv ! nvv4l2h265enc control-rate=1 qp-range=\"28,50:0,38:0,50\" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux.      alsasrc ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! audioconvert ! opusenc bitrate=320000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux.     mpegtsmux name=mux ! appsink"
6

command not found for this.

Did you mean: gst-launch-1.0 -v ?

7

Oh…yes ! I’ll edit…sorry

8

So I ran it and I get these errors until it quits:

Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
Error: Can't initialize nvrm channel
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
WARNING: erroneous pipeline: no element "nvv4l2decoder"
9

I suppose you face that when trying to run first pipeline for viewing cam1. Better show the exact command you run with output.

Do you have a full JetPack install ? nvv4l2decoder plugin should be installed from package nvidia-l4t-gstreamer.
What gives:

sudo apt update
sudo apt search nvidia-l4t-gstreamer
10

All packages are up to date.
b3ck@belabox:~$ sudo apt search nvidia-l4t-gstreamer
Sorting… Done
Full Text Search… Done
nvidia-l4t-gstreamer/stable,now 32.6.1-20210916211029 arm64 [installed]
NVIDIA GST Application files

b3ck@belabox:~$

11

May be a memory issue… I have no Nano for testing.

12

the default pipeline for webcams works and it has it in there:

v4l2src ! image/jpeg ! 
identity name=v_delay signal-handoffs=TRUE ! 
nvv4l2decoder mjpeg=1 enable-max-performance=true ! nvvidconv ! 
textoverlay text='' valignment=top halignment=right font-desc="Monospace, 5" name=overlay ! queue ! 
nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 
alsasrc device=hw:2 ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! voaacenc bitrate=128000 ! aacparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux. 
mpegtsmux name=mux ! 
appsink name=appsink
13

hmmm, I have done it with a camlink HDMI source and a webcam at once, and it worked, on the same jetson nano, I just can’t find that pipeline I created otherwise I would use that as a starting point =(

14

It may be an issue with USB camera driver reserving all available bandwidth with MJPG… Nano shares the same controller for USB ports. Searching this forum you may find some info, but I have no personal experience with this case, so unable to give some other advice.

1 Like
15

okay thank you for trying, I really appreciate it!

16

Sorry, I don’t have even one MJPG cam, so just tried simulting these.
You may try:

gst-launch-1.0 -v \
  v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink  \
  v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink

If this fails, probably it is related to USB issue.

17 
b3ck@belabox:~$ gst-launch-1.0 -v \
>   v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink  \
>   v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink
WARNING: erroneous pipeline: no element "nvv4l2decoder"
b3ck@belabox:~$
18

which is weird because as you can see here in this screenshot, I am showing the pipeline which uses nvv4l2decoder and it’s working with the webcam:

Screen Shot 2022-09-08 at 1.01.16 PM
Screen Shot 2022-09-08 at 1.01.16 PM1920×1149 133 KB

19

Do you see some kernel errors when this happens ? Launch in another terminal:

sudo dmesg --follow

And relaunch the gst-launch command. If you see errors, you may post these here.

Also be sure that each camera uses a native mode, such as the ones listed by :

v4l2-ctl -d0 --list-formats-ext
v4l2-ctl -d1 --list-formats-ext

In other words, does these command work for reading each camera ?

gst-launch-1.0 -v  v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink  

gst-launch-1.0 -v  v4l2src device=/dev/video1 ! image/jpeg,width=800,height=448,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink
20

you want me to post my whole dmesg here?