"No control in path" using nvcompositor to merge 4 webcams into an RTSP stream

Hi there,

I have a 4Gb Jetson Nano and am trying to combine 4 webcams to stream using python / gstreamer to a more powerful machine with an RTX 2060 graphics card for processing using a Pytorch model.

The 4 webcams are identical. The video from each of them needs to be at 1920x1080 and 30fps. The webcams are only capable of serving at that rate using MJPEG.

My aim is to use the GPU in the Nano to take the load of merging the streams and encoding them in H264/5 to reduce the network traffic.

I am therefore attempting to use nvcompositor to merge the webcams, nvv4l2h264enc for encoding and rtph264pay to stream the video, but I can’t get it to work.

My python script is as follows:

import gi
import os

gi.require_version('Gst', '1.0')
gi.require_version('GstRtspServer', '1.0')
from gi.repository import Gst, GLib, GstRtspServer

os.environ['GST_DEBUG'] = '3'

Gst.init(None)

port = "8554"
mount_point = "/test"

server = GstRtspServer.RTSPServer.new()
server.set_service(port)
mounts = server.get_mount_points()
factory = GstRtspServer.RTSPMediaFactory.new()

width=1920
height=1080
factory.set_launch(f'nvcompositor name=comp  sink_0::xpos=0       sink_0::ypos=0        sink_0::width={width} sink_0::height={height} '
                   f'                        sink_1::xpos={width} sink_1::ypos=0        sink_1::width={width} sink_1::height={height} '
                   f'                        sink_2::xpos=0       sink_2::ypos={height} sink_2::width={width} sink_2::height={height} '
                   f'                        sink_3::xpos={width} sink_3::ypos={height} sink_3::width={width} sink_3::height={height} '
                   f'! queue '
                   f'! nvvidconv '
                   f'! video/x-raw(memory:NVMM), format=NV12, width={width * 2}, height={height * 2}, framerate=30/1 '
                   f'! nvv4l2h264enc '
                   f'! rtph264pay '
                   f'v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1920,height=1080 ! nvv4l2decoder mjpeg=1 ! comp.sink_0 '
                   f'v4l2src device=/dev/video1 io-mode=2 ! image/jpeg,width=1920,height=1080 ! nvv4l2decoder mjpeg=1 ! comp.sink_1 '
                   f'v4l2src device=/dev/video2 io-mode=2 ! image/jpeg,width=1920,height=1080 ! nvv4l2decoder mjpeg=1 ! comp.sink_2 '
                   f'v4l2src device=/dev/video3 io-mode=2 ! image/jpeg,width=1920,height=1080 ! nvv4l2decoder mjpeg=1 ! comp.sink_3 ')

mounts.add_factory(mount_point, factory)
server.attach()

print ("stream ready at rtsp://127.0.0.1:" + port + "/test")

loop = GLib.MainLoop()
loop.run()

My problem is that when I run the script and try to connect to it using VLC all I get for output is the following:

stream ready at rtsp://127.0.0.1:8554/test
0:00:02.781204229   238   0x55816bfa30 FIXME              rtspmedia rtsp-media.c:3841:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:02.827036565   238   0x7fa80d0f20 FIXME              rtspmedia rtsp-media.c:3841:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:02.827096202   238   0x7fa80d0f20 ERROR             rtspclient rtsp-client.c:2651:handle_setup_request: client 0x55816c8260: no control in path '/test'

I’ve seen a couple of other threads that relate to the ‘no control in path’ message, but I haven’t been able to use them to solve my issue.

The machine is running R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t210ref, EABI: aarch64, DATE: Fri Oct 16 19:44:43 UTC 2020

The python script is running in a docker container based on nvcr.io/nvidia/l4t-base:r32.5.0

The dockerfile is as follows:

FROM nvcr.io/nvidia/l4t-base:r32.5.0

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get -y -qq update && apt-get install -y -qq xz-utils build-essential zlib1g-dev
RUN apt-get install -y -qq libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev

WORKDIR "/tmp"
RUN wget --quiet --no-check-certificate https://www.python.org/ftp/python/3.8.10/Python-3.8.10.tar.xz && \
    tar xf Python-3.8.10.tar.xz && \
    cd Python-3.8.10 && ls -l && \
    ./configure --enable-optimizations && \
    make && make install

WORKDIR /opt/venv
RUN python3 -m venv /opt/venv && \
    . /opt/venv/bin/activate && \
    python3 -m pip install --upgrade pip

RUN apt-get update && \
    apt-get install -y libgirepository1.0-dev libcairo2-dev gir1.2-gstreamer-1.0 pkg-config python3-dev gir1.2-gtk-3.0 gir1.2-gst-rtsp-server-1.0 v4l-utils

WORKDIR /opt/CamStreamer
COPY ./ .
ENTRYPOINT [ "/opt/CamStreamer/entrypoint.sh" ]

RUN . /opt/venv/bin/activate && pip install -r requirements.txt

To run the docker container I use:

docker run --rm -it -p 8554:8554 -v "${THIS_DIR}/../app":/opt/CamStreamer \
		--device /dev/video0 \
		--device /dev/video1 \
		--device /dev/video2 \
		--device /dev/video3 \
		cam-streamer "$@"

Any help would be very much appreciated.

Archie

Maybe you just need to add name=pay0 into rtph264pay properties.

For reference, the following adapted from your code works fine with R32.5.1 on my NX:

#!/usr/bin/env python

import gi
import os

gi.require_version('Gst', '1.0')
gi.require_version('GstRtspServer', '1.0')
from gi.repository import Gst, GLib, GstRtspServer

os.environ['GST_DEBUG'] = '3'

Gst.init(None)

port = "8554"
mount_point = "/test"

server = GstRtspServer.RTSPServer.new()
server.set_service(port)
mounts = server.get_mount_points()
factory = GstRtspServer.RTSPMediaFactory.new()

width=1920
height=1080
factory.set_launch('nvcompositor name=comp  sink_0::xpos=0       sink_0::ypos=0        sink_0::width=960     sink_0::height=540     \
                                            sink_1::xpos=960     sink_1::ypos=0        sink_1::width=960     sink_1::height=540     \
                                            sink_2::xpos=0       sink_2::ypos=540      sink_2::width=960     sink_2::height=540     \
                                            sink_3::xpos=960     sink_3::ypos=540      sink_3::width=960     sink_3::height=540     \
                   ! queue \
                   ! nvvidconv \
                   ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 \
                   ! nvv4l2h264enc insert-vui=1 \
                   ! rtph264pay name=pay0 \
                   videotestsrc ! video/x-raw,width=480,height=270,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=1920,height=1080, format=NV12 ! comp.sink_0 \
                   videotestsrc ! video/x-raw,width=480,height=270,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=1920,height=1080, format=NV12 ! comp.sink_1 \
                   videotestsrc ! video/x-raw,width=480,height=270,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=1920,height=1080, format=NV12 ! comp.sink_2 \
                   videotestsrc ! video/x-raw,width=480,height=270,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=1920,height=1080, format=NV12 ! comp.sink_3 ')

mounts.add_factory(mount_point, factory)
server.attach()

print ("stream ready at rtsp://127.0.0.1:" + port + "/test")

loop = GLib.MainLoop()
loop.run()

Hi Honey_Patouceul - many many thanks for trying this out and for identifying the issue. I had previously had name=pay0 in there but took it out for some reason.

So now I have the situation where it is streaming the video, but there is something still going wrong. There are banded areas of the image where the colours are all screwed up and some parts of the image have ghosts of other parts of the image.

I see that you have tweaked the pipeline a little bit to pre-process the streams before passing them to the nvcompositor. I’ll try updating mine to take the same approach and let you know if that makes any difference.

If you have any other suggestions as to how I might fix the colour issue in the meantime I’d be very grateful !

Cheers,
Archie

For the banded parts, if these are black, it might be a pixel-aspect-ratio or frame-aspect-ratio or else issue with the display sink adjusting. I’d suggest you be sure about the resolution of each nvcompositor source and output, you may use nvvidconv before each input for setting width and height, also add pixel-aspect-ratio=1/1 into output caps when rescaling.

If it is a color issue, you may try to convert each input into RGBA (NVMM) and have nvcompositor to output in RGBA (NVMM) as well for trying, you would use nvvidconv later if other format is needed.

[EDIT:
Additional notes:

  • it may be better to add a queue before each nvcompositor sink
  • I adjusted your pipeline for reproducing your case, but displaying with nvoverlaysink and a low height monitor, so reduced the layout. Obviously it has no sense to rescale into 1080p with nvvidconv and then rescale down into 540p in nvcompositor.

]

Hi again Honey_Patouceul,

I got it working !

The issue around the wierd colours seems to be related to having nvv4l2decoder mjpeg=1. When I removed the mjpeg=1 part that fixed that and the colours appear normal now. It seems from a bit of googling that there is an issue with that setting.

However, it didn’t solve all my problems. I was then getting the following:

Opening in BLOCKING MODE 
0:00:02.143411598   615   0x7f8810c4a0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x7f880f30b0 Failed to determine interlace mode
0:00:02.143524100   615   0x7f8810c4a0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x7f880f30b0 Failed to determine interlace mode
0:00:02.143590612   615   0x7f8810c4a0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x7f880f30b0 Failed to determine interlace mode
0:00:02.143707281   615   0x7f8810c4a0 WARN                    v4l2 gstv4l2object.c:4430:gst_v4l2_object_probe_caps:<nvv4l2h264enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
Opening in BLOCKING MODE 
0:00:02.146642809   615   0x7f8810c4a0 WARN                    v4l2 gstv4l2object.c:4430:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:02.146710779   615   0x7f8810c4a0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x7f880ea020 Failed to determine interlace mode
NvMMLiteBlockCreate : Block : BlockType = 256 
0:00:02.179946027   615   0x7f80030000 WARN          v4l2bufferpool gstv4l2bufferpool.c:790:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold
[JPEG Decode] BeginSequence Display WidthxHeight 1920x1080
0:00:02.768191276   615   0x7f80030000 WARN                    v4l2 gstv4l2object.c:4430:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:02.768269611   615   0x7f80030000 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x7f880ea020 Failed to determine interlace mode
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
0:00:02.780902468   615   0x7f80030000 WARN          v4l2bufferpool gstv4l2bufferpool.c:1057:gst_v4l2_buffer_pool_start:<nvv4l2h264enc0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:02.782511406   615   0x7f80030000 WARN            v4l2videodec gstv4l2videodec.c:1632:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:02.788999765   615   0x7f80030000 WARN          v4l2bufferpool gstv4l2bufferpool.c:1057:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:02.826379523   615   0x7f70002850 WARN          v4l2bufferpool gstv4l2bufferpool.c:1503:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
H264: Profile = 66, Level = 0 
0:00:02.919887173   615   0x7f60007800 WARN          v4l2bufferpool gstv4l2bufferpool.c:1503:gst_v4l2_buffer_pool_dqbuf:<nvv4l2h264enc0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:02.927491804   615   0x557c16f630 FIXME              rtspmedia rtsp-media.c:3841:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:02.933111083   615   0x557c16f630 FIXME              rtspmedia rtsp-media.c:3841:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:02.933189887   615   0x557c16f630 WARN               rtspmedia rtsp-media.c:3867:gst_rtsp_media_suspend: media 0x7f881151d0 was not prepared
0:00:02.943796505   615   0x557c16f630 FIXME             rtspclient rtsp-client.c:1657:handle_play_request:<GstRTSPClient@0x557c178150> Add support for seek style (null)
0:00:02.944108178   615   0x557c16f630 FIXME              rtspmedia rtsp-media.c:2437:gst_rtsp_media_seek_full:<GstRTSPMedia@0x7f881151d0> Handle going back to 0 for none live not seekable streams.
0:00:02.944188388   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.958628573   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.960120165   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.960206938   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.960364805   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.960548871   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.983345942   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.983428496   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.983472142   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.983526050   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:02.983571259   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:03.004651316   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:03.004808506   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
0:00:03.004885852   615   0x7f70002850 WARN            v4l2videodec gstv4l2videodec.c:1217:gst_v4l2_video_dec_loop:<nvv4l2decoder0> Decoder is producing too many buffers
...

This seems to be some issue with nvv4l2decoder that I can’t quite understand. The solution was to replace this with jpegdec and the ‘too many buffers’ warning went away.

So the final working pipeline is as follows:

nvcompositor name=comp sink_0::xpos=0    sink_0::ypos=0    sink_0::width=1920 sink_0::height=1080
                       sink_1::xpos=1920 sink_1::ypos=0    sink_1::width=1920 sink_1::height=1080
                       sink_2::xpos=0    sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080
                       sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 
! nvvidconv 
! video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=NV12 
! nvv4l2h264enc preset-level=4 insert-vui=1 iframeinterval=5 
! rtph264pay name=pay0 
v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=30/1,pixel-aspect-ratio=1/1 ! jpegdec ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=RGBA ! queue ! comp.sink_0 
v4l2src device=/dev/video1 ! image/jpeg,width=1920,height=1080,framerate=30/1,pixel-aspect-ratio=1/1 ! jpegdec ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=RGBA ! queue ! comp.sink_1 
v4l2src device=/dev/video2 ! image/jpeg,width=1920,height=1080,framerate=30/1,pixel-aspect-ratio=1/1 ! jpegdec ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=RGBA ! queue ! comp.sink_2 
v4l2src device=/dev/video3 ! image/jpeg,width=1920,height=1080,framerate=30/1,pixel-aspect-ratio=1/1 ! jpegdec ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=RGBA ! queue ! comp.sink_3

Thank you so much for your help I don’t think I’d ever have solved it without it.

To be honest, I’ve been writing code for over 35 years from assembler, C, C++, Java, JavaScript, Lisp, Python, SQL and best of all Ruby, but in all that time and all the frameworks I’ve used, GStreamer is the most opaque, difficult one I’ve ever come across.

Much of the time it seems like black magic whether a pipeline will work or not and while conceptually it kinda makes sense, you spend 99% of your time trying different things to get it to work, trying to work out what some obscure error means and googling to see if someone else has succeeded in doing what you want to do.

I hope that this little issue and pipeline here will save someone else a bunch of time.

Cheers,
Archie

You may try this similar pipeline leveraging nvjpegdec, it may save CPU load as compared to jpegdec:

nvcompositor name=comp sink_0::xpos=0    sink_0::ypos=0    sink_0::width=1920 sink_0::height=1080
                       sink_1::xpos=1920 sink_1::ypos=0    sink_1::width=1920 sink_1::height=1080
                       sink_2::xpos=0    sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080
                       sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 
! nvvidconv 
! video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=NV12 
! nvv4l2h264enc preset-level=4 insert-vui=1 iframeinterval=5 
! rtph264pay name=pay0 
v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_0 
v4l2src device=/dev/video1 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_1 
v4l2src device=/dev/video2 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_2 
v4l2src device=/dev/video3 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_3

maybe the NV12(NVMM) to RGBA(NVMM) conversion of inputs is not required, you may also try without it.

I’d also be curious about the jpeg format that your camera/driver sends. Assuming you have apt package v4l-utils installed, you may post the output of (for 1st camera on /dev/video0):

v4l2-ctl -d0 --list-formats-ext

Hi there,

I tried swapping jpegdec for nvjpegdec, but it caused a segmentation fault. It’s a typical example of why gstreamer is so frustrating. You’d expect given the names of them that one would be a drop-in replacement for the other, but no it isn’t.

On closer inspection, jpegdec accepts the following combinations:

  • video/x-raw with formats I420, RGB, BGR, RGBx, xRGB, BGRx, xBGR or GRAY8

However, nvjpegdec only accepts the following:

  • video/x-raw with formats I420, RGB, BGR, RGBx, xRGB, BGRx, xBGR, GRAY8
  • video/x-raw(memory:NVMM) with formats I420 or GRAY8

Here is the output from v4l2-ctl -d0 --list-formats-ext

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'MJPG' (compressed)
	Name        : Motion-JPEG
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1600x896
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1024x768
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1024x576
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 960x720
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 960x544
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 864x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 800x600
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 800x448
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 352x288
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1600x896
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 1024x768
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 1024x576
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 960x720
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 960x544
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 864x480
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 800x600
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 800x448
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 352x288
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.200s (5.000 fps)

As you can see theYUYV format only outputs at 5fps @ 1920x1080 so it’s no good to me and I have to use MJPG.

In regards to the RGBA conversion, I have to do that because nvcompositor only accepts RGBA.

And for the NV12 conversion, that is necessary because nvv4l2h264enc only accepts video/x-raw(memory:NVMM) with formats I420 or NV12 or P010_10LE.

In terms of performance it mostly works ok with jpegdec. One thing I have noticed is that with my working pipeline above, the GPU sits at 0% activity (using jtop), so it seems my effort to delegate to that isn’t really working.

Additionally, I do get regular dropped frames from individual webcams (not all of them simultaneously) and I’m not sure why. My assumption / hope was that gstreamer would setting up multiple threads for each of the webcams, but I don’t know for sure or how to check that.

Cheers,
Archie

nvcompositor only outputs in RGBA format, but it can use also use NV12 or I420 memory as input.
You may check for your version with:

gst-inspect-1.0 nvcompositor

and check for SINK template.

You may also set format MJPG in jpeg caps:

nvcompositor name=comp sink_0::xpos=0    sink_0::ypos=0    sink_0::width=1920 sink_0::height=1080
                       sink_1::xpos=1920 sink_1::ypos=0    sink_1::width=1920 sink_1::height=1080
                       sink_2::xpos=0    sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080
                       sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 
! nvvidconv 
! video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=NV12 
! nvv4l2h264enc preset-level=4 insert-vui=1 insert-sps-pps=1 iframeinterval=5 
! rtph264pay name=pay0 
v4l2src device=/dev/video0 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_0 
v4l2src device=/dev/video1 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_1 
v4l2src device=/dev/video2 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_2 
v4l2src device=/dev/video3 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! comp.sink_3

# Just using I420 as nvcompositor input
nvcompositor name=comp sink_0::xpos=0    sink_0::ypos=0    sink_0::width=1920 sink_0::height=1080
                       sink_1::xpos=1920 sink_1::ypos=0    sink_1::width=1920 sink_1::height=1080
                       sink_2::xpos=0    sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080
                       sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 
! nvvidconv 
! video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=NV12 
! nvv4l2h264enc preset-level=4 insert-vui=1 insert-sps-pps=1 iframeinterval=5 
! rtph264pay name=pay0 
v4l2src device=/dev/video0 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! queue ! comp.sink_0 
v4l2src device=/dev/video1 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! queue ! comp.sink_1 
v4l2src device=/dev/video2 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! queue ! comp.sink_2 
v4l2src device=/dev/video3 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw(memory:NVMM),format=I420 ! queue ! comp.sink_3

# Or using NV12 inputs from nvv4l2decoder
nvcompositor name=comp sink_0::xpos=0    sink_0::ypos=0    sink_0::width=1920 sink_0::height=1080
                       sink_1::xpos=1920 sink_1::ypos=0    sink_1::width=1920 sink_1::height=1080
                       sink_2::xpos=0    sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080
                       sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 
! nvvidconv 
! video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=NV12 
! nvv4l2h264enc preset-level=4 insert-vui=1 insert-sps-pps=1 iframeinterval=5 
! rtph264pay name=pay0 
v4l2src device=/dev/video0 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvv4l2decoder mjpeg=1 ! video/x-raw(memory:NVMM),format=NV12 ! queue ! comp.sink_0 
v4l2src device=/dev/video1 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvv4l2decoder mjpeg=1 ! video/x-raw(memory:NVMM),format=NV12 ! queue ! comp.sink_1 
v4l2src device=/dev/video2 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvv4l2decoder mjpeg=1 ! video/x-raw(memory:NVMM),format=NV12 ! queue ! comp.sink_2 
v4l2src device=/dev/video3 ! image/jpeg,format=MJPG,width=1920,height=1080,framerate=30/1 ! nvv4l2decoder mjpeg=1 ! video/x-raw(memory:NVMM),format=NV12 ! queue ! comp.sink_3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.