RTSP Gstreamer pipeline with nvv4l2h264

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

I’m running a gstreamer pipeline with nvcompositor to show multiple cameras simultaneously.

My pipeline previously ran with omx264enc and worked well, however it looks like on a new device (orin AGX) there are no omx plugins anymore. I’ve switched my pipeline to use x264enc and this is working, but it’s extremely slow and unusable:

./test-launch  "( nvcompositor name=comp \
    sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720 \
    sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 \
    sink_2::xpos=2560 sink_2::ypos=0 sink_2::width=1280 sink_2::height=720 \
    sink_3::xpos=0 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720 \
    sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720 \
    sink_5::xpos=2560 sink_5::ypos=720 sink_5::width=1280 sink_5::height=720 \
    ! nvvidconv ! x264enc ! rtph264pay name=pay0 pt=96 \
    v4l2src device=/dev/video0 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video1 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video2 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video3 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video4 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video5 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    )

I’m attempting to switch in nvv4l2h264enc but haven’t yet been successful. Here’s the pipeline I’m testing (I’ve tried with both v4l2src as well as nvv4l2camerasrc but neither work.

./test-launch "( nvcompositor name=comp \
    sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720 \
    sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 \
    sink_2::xpos=2560 sink_2::ypos=0 sink_2::width=1280 sink_2::height=720 \
    sink_3::xpos=0 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720 \
    sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720 \
    ! nvvidconv ! nvv4l2h264enc ! video/x-h264, profile=baseline, stream-format=byte-stream ! h264parse ! rtph264pay name=pay0 pt=96 \
    v4l2src device=/dev/video0 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video1 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video2 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video3 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    v4l2src device=/dev/video4 ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! comp. \
    ) "

Does anyone have any ideas?

3

You may add a framerate according to what your cameras can provide. Check the available framerates and resolutions with :

# v4l2-ctl command is provided by apt package v4l-utils
sudo apt install v4l-utils

# Get your camera available modes (assuming here that your camera is /dev/video1)
v4l2-ctl -d /dev/video1 --list-formats-ext

Then pick one supported mode and adjust from the following if it works for your case:

./test-launch "\
videotestsrc ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=(string)RGBA ! comp.sink_0 \
videotestsrc ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=(string)RGBA ! comp.sink_1 \
videotestsrc ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=(string)RGBA ! comp.sink_2 \
videotestsrc ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=(string)RGBA ! comp.sink_3 \
videotestsrc ! video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=(string)RGBA ! comp.sink_4 \
nvcompositor name=comp  \
   sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720  \
   sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720  \
   sink_2::xpos=2560 sink_2::ypos=0   sink_2::width=1280 sink_2::height=720  \
   sink_3::xpos=0    sink_3::ypos=720 sink_3::width=1280 sink_3::height=720  \
   sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720  \
 ! video/x-raw(memory:NVMM),format=RGBA,width=(int)3840,height=(int)1440,framerate=(fraction)30/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12 ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 insert-vui=1 ! h264parse ! rtph264pay name=pay0 "

Then you should be able to read it from Jetson (assuming it has a display) with:

gst-play-1.0 rtsp://127.0.0.1:8554/test

# Or setting latency to 100ms:
gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test latency=100 ! rtph264depay ! decodebin ! nvvidconv ! autovideosink

If ok, then try from another LAN host.

6

Hi,
Please try Honey Patouceul’s suggestion. And you may confirm the pipeline is valid in gst-launch-1.0 command with fakesink. For example, if we want to run

$ ./test-launch "videotestsrc is-live=1 ! nvvidconv ! nvv4l2h264enc ! h264parse ! rtph264pay name=pay0 pt=96"

We can run this command beforehand to make sure the pipeline is good:

$ gst-launch-1.0 videotestsrc is-live=1 ! nvvidconv ! nvv4l2h264enc ! h264parse ! rtph264pay name=pay0 pt=96 ! fakesink
7

Thanks @DaneLLL and @Honey_Patouceul

Running this with fakesink works fine:

gst-launch-1.0 \
v4l2src device=/dev/video0 ! "video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)RGBA" ! comp.sink_0 \
v4l2src device=/dev/video1 ! "video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)RGBA" ! comp.sink_1 \
v4l2src device=/dev/video2 ! "video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)RGBA" ! comp.sink_2 \
v4l2src device=/dev/video3 ! "video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)RGBA" ! comp.sink_3 \
v4l2src device=/dev/video4 ! "video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)RGBA" ! comp.sink_4 \
nvcompositor name=comp  \
   sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720  \
   sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720  \
   sink_2::xpos=2560 sink_2::ypos=0   sink_2::width=1280 sink_2::height=720  \
   sink_3::xpos=0    sink_3::ypos=720 sink_3::width=1280 sink_3::height=720  \
   sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720  \
 ! "video/x-raw(memory:NVMM),format=RGBA,width=(int)3840,height=(int)1440,framerate=(fraction)30/1" ! nvvidconv ! "video/x-raw(memory:NVMM), format=NV12" ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 insert-vui=1 ! h264parse ! rtph264pay name=pay0 ! fakesink

However when I switch to test-launch or gst-rtsp-launch I get these messages that ArgusV4L2_Open failed and the stream doesn’t work:

stream ready at rtsp://127.0.0.1:8554/test
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
H264: Profile = 66, Level = 0
NVMEDIA: Need to set EMC bandwidth : 2872000
Opening in BLOCKING MODE
Opening in BLOCKING MODE
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE
Opening in BLOCKING MODE
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE
Opening in BLOCKING MODE
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE
Opening in BLOCKING MODE
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE
Opening in BLOCKING MODE
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
H264: Profile = 66, Level = 0
NVMEDIA: Need to set EMC bandwidth : 2872000

I’ve tried replacing v4l2src with nvv4l2camerasrc and that’s still not working, for example:

./test-launch "\
nvv4l2camerasrc device=/dev/video0 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)I420 ! comp.sink_0 \
nvv4l2camerasrc device=/dev/video1 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)I420 ! comp.sink_1 \
nvv4l2camerasrc device=/dev/video2 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)I420 ! comp.sink_2 \
nvv4l2camerasrc device=/dev/video3 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)I420 ! comp.sink_3 \
nvv4l2camerasrc device=/dev/video4 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)I420 ! comp.sink_4 \
nvcompositor name=comp  \
   sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720  \
   sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720  \
   sink_2::xpos=2560 sink_2::ypos=0   sink_2::width=1280 sink_2::height=720  \
   sink_3::xpos=0    sink_3::ypos=720 sink_3::width=1280 sink_3::height=720  \
   sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720  \
 ! video/x-raw(memory:NVMM),format=RGBA,width=(int)3840,height=(int)1440,framerate=(fraction)30/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12 ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 insert-vui=1 ! h264parse ! rtph264pay name=pay0 "

This works with gst-launch-1.0 and a fakesink but causes a segfault with test-launch

The devices are definitely available:

$ ls /dev/video*
/dev/video0  /dev/video1  /dev/video2  /dev/video3  /dev/video4

and

$v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'UYVY' (UYVY 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 3840x2160
			Interval: Discrete 0.062s (16.000 fps)
	[1]: 'NV16' (Y/CbCr 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 3840x2160
			Interval: Discrete 0.062s (16.000 fps)
8

Try setting a native mode of your camera such as 1280x720@60fps and keep this framerate for the whole pipeline;

./test-launch "\
nvv4l2camerasrc device=/dev/video0 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! queue ! comp.sink_0 \
nvv4l2camerasrc device=/dev/video1 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! queue ! comp.sink_1 \
nvv4l2camerasrc device=/dev/video2 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! queue ! comp.sink_2 \
nvv4l2camerasrc device=/dev/video3 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! queue ! comp.sink_3 \
nvv4l2camerasrc device=/dev/video4 ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720, framerate=60/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! queue ! comp.sink_4 \
nvcompositor name=comp  \
   sink_0::xpos=0    sink_0::ypos=0   sink_0::width=1280 sink_0::height=720  \
   sink_1::xpos=1280 sink_1::ypos=0   sink_1::width=1280 sink_1::height=720  \
   sink_2::xpos=2560 sink_2::ypos=0   sink_2::width=1280 sink_2::height=720  \
   sink_3::xpos=0    sink_3::ypos=720 sink_3::width=1280 sink_3::height=720  \
   sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720  \
 ! video/x-raw(memory:NVMM),format=RGBA,width=(int)3840,height=(int)1440,framerate=(fraction)60/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12 ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 insert-vui=1 ! h264parse ! rtph264pay name=pay0 "
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.