Stream multiple cameras with nvmultistreamtiler via udpink/udpsrc

Sealfoss · May 17, 2023, 9:21pm

I am on an Xavier NX with Jetpack 5.0.2.

I’m attempting to stream multiple cameras over a single video stream with nvmultistreamtiler. I intend to send the stream via udpsink and receive via udpsrc on the client. Here is my example pipe line string for how I am trying to send the video stream:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 tnr-mode=2 tnr-strength=1 wbmode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)NV12,framerate=(fraction)30/1' ! m.sink_0 nvarguscamerasrc sensor-id=1 tnr-mode=2 tnr-strength=1 wbmode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)NV12,framerate=(fraction)30/1' ! m.sink_1 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvmultistreamtiler width=1920 height=1080 interpolation-method=2 rows=1 columns=2 ! queue ! nvv4l2h264enc iframeinterval=1 ! queue ! rtph264pay pt=96 config-interval=1 ! udpsink host=192.168.1.171 port=7331 sync=0

On the client end, here is how I’m attempting to receive and display the video stream:

gst-launch-1.0 -v udpsrc port=7331 ! application/x-rtp,media=video,encoding=H264,payload=96 ! rtph264depay ! avdec_h264 ! autovideosink sync=0

I am able to send/receive a single camera just fine via methods similar to the above. However, once I try to use multistreamtiler, the stream essentially sends a single frame and dies. Here is the output from the server:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 tnr-mode=2 tnr-strength=1 wbmode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)NV12,framerate=(fraction)30/1' ! m.sink_0 nvarguscamerasrc sensor-id=1 tnr-mode=2 tnr-strength=1 wbmode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)NV12,framerate=(fraction)30/1' ! m.sink_1 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvmultistreamtiler width=1920 height=1080 interpolation-method=2 rows=1 columns=2 ! queue ! nvv4l2h264enc iframeinterval=1 ! queue ! rtph264pay pt=96 config-interval=1 ! udpsink host=192.168.1.171 port=7331 sync=0
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
Redistribute latency...
NvMMLiteBlockCreate : Block : BlockType = 4 
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3280 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3280 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 3280 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3280 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
NVMEDIA: Need to set EMC bandwidth : 846000 
NVMEDIA_ENC: bBlitMode is set to TRUE

On the receive side, I see a single frame of video, along with the following text output on the command line:

reed@motherbrain:~$ gst-launch-1.0 -v udpsrc port=7331 ! application/x-rtp,media=video,encoding=H264,payload=96 ! rtph264depay ! avdec_h264 ! autovideosink sync=0
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = application/x-rtp, media=(string)video, encoding=(string)H264, payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, encoding=(string)H264, payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000b67424028965403c0113f2a01000468ce3c80, level=(string)4, profile=(string)constrained-baseline
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000b67424028965403c0113f2a01000468ce3c80, level=(string)4, profile=(string)constrained-baseline
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstXvImageSink:autovideosink0-actual-sink-xvimage.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1
Redistribute latency...

I have gotten similar pipleines to work in the past on Nvidia Jetson, however that was a different version of JetPack and involved the bufapi options on nvarguscamerasrc which don’t exist in Jetpack 5.02.

Any help is appreciated, thanks in advance.

Honey_Patouceul · May 18, 2023, 12:52am

On receiver side, try changing caps:

gst-launch-1.0 -v udpsrc port=7331 ! application/x-rtp,media=video,encoding=H264,...

into

gst-launch-1.0 -v udpsrc port=7331 ! application/x-rtp,media=video,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! ...

Sealfoss · May 18, 2023, 1:57am

Adding the h264parse didn’t fix the problem. No change.

DaneLLL · May 18, 2023, 3:57am

Hi,
The latest Jetpack 5 release is 5.1.1. you may upgrade to latest release and try again. Besides, you can use nvcompositor plugin for this use-case.

Sealfoss · May 18, 2023, 1:26pm

I don’t think I can use the latest jetpack due to driver incompatibility w/ my carrier board (Seeed a205), unfortunately. I’m going to check out nvcompositer now thank you.

Honey_Patouceul · May 18, 2023, 1:28pm

Did you change caps ? In particular change encoding into encoding-name.

Sealfoss · May 18, 2023, 1:30pm

I actually did miss that in your proposed pipeline, but just tried it again with “encoding-name” instead and it failed the same way it had previously.

Sealfoss · May 18, 2023, 2:04pm

Ok, I swapped out nvmultistreamtiler with nvcompositor and it works now as intended. In fact, it works way better, there had been significant latency (~500ms) when using multistreamtiler to tile the streams just for viewing, let alone transport across the network.

Here is the pipeline I’m using successfully now:

gst-launch-1.0 \
nvcompositor name=comp  \
sink_0::xpos=1920 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_0::alpha=1 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 sink_1::zorder=2 sink_1::alpha=0.1 ! \
'video/x-raw(memory:NVMM),format=RGBA' ! \
nvvidconv ! \
nvv4l2h264enc iframeinterval=1 insert-sps-pps=1 idrinterval=30 insert-vui=1 ! \
rtph264pay pt=96 config-interval=1 ! \
udpsink host=192.168.1.171 port=7331 sync=0 \
nvarguscamerasrc sensor-id=1 tnr-mode=2 tnr-strength=1 wbmode=0 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)NV12,framerate=(fraction)30/1' ! \
nvvidconv ! \
'video/x-raw(memory:NVMM),format=RGBA' ! \
comp.sink_0    \
nvarguscamerasrc sensor-id=0 tnr-mode=2 tnr-strength=1 wbmode=0 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)NV12,framerate=(fraction)30/1' ! \
nvvidconv ! \
'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080' ! \
comp.sink_1

system · June 7, 2023, 6:17am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Change video settings on RTSP streaming Jetson Nano gstreamer	15	4587	March 9, 2022
Multiple stream from one camera via gstreamer Jetson Nano camera , gstreamer	11	6452	October 15, 2021
GStreamer python script to send and receive UDP stream Jetson AGX Xavier gstreamer , python	10	4761	October 18, 2021
Quark Carrier Gstreamer UDP video stream problem Jetson Xavier NX gstreamer	4	1091	December 8, 2021
Deepstream crashes with camera source, but works fine with nvgstcapture DeepStream SDK	6	716	October 12, 2021
Deepstream 3.0 C API Jetson AGX Xavier	10	2487	May 30, 2019
Deepstream Pipeline with NVOF and NVOFVISUAL Elements on Jetson Orin Nano DeepStream SDK	18	270	June 17, 2024
Save 4 usb camera stream to 1 concatenated video Jetson Xavier NX camera , usb	20	1776	October 18, 2021
Problems with streaming ( and receiving ) a simple usb-camera with jetson nano and gstreamer Jetson Nano camera , gstreamer	9	4302	October 15, 2021
Python binding send RTP DeepStream SDK	13	387	June 15, 2023

Stream multiple cameras with nvmultistreamtiler via udpink/udpsrc

Related topics