Composite Two CSI Camera stream into single RTP stream

Casmer · June 11, 2020, 7:35pm

I am trying to combine both CSI Camera streams into a single sidebyside stream to send to my remote pc to view on a stereoscopic vr headset, but I can’t get the stream to send convert to h264 encoding.
My pipeline call looks like this:

WIDTH=1280
HEIGHT=720
WIDTH2=2560
HEIGHT2=720
CAPS=“video/x-raw, width=$WIDTH, height=$HEIGHT”
CLIENT_IP=192.168.0.30

gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=$WIDTH sink_0::height=$HEIGHT sink_1::xpos=$WIDTH sink_1::ypos=0 sink_1::width=$WIDTH sink_1::height=$HEIGHT ! “video/x-raw,width=$WIDTH2, height=$HEIGHT2, framerate=21/1” ! nvvidconv ! nvv4l2h264enc bitrate=8000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false nvarguscamerasrc name=camsrc0 sensor_id=0 ! ‘video/x-raw(memory:NVMM),width=1280, height=720, framerate=21/1, format=NV12’ ! nvvidconv flip-method=0 ! “video/x-raw,width=$WIDTH, height=$HEIGHT” ! comp. nvarguscamerasrc name=camsrc1 sensor_id=1 ! ‘video/x-raw(memory:NVMM),width=1280, height=720, framerate=21/1, format=NV12’ ! nvvidconv flip-method=0 ! “video/x-raw,width=$WIDTH, height=$HEIGHT” ! comp.

And the ouptut i get is this:

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
Redistribute latency...
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:camsrc1: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvArgusCameraSrc:camsrc1:
streaming stopped, reason not-negotiated (-4)
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1 
   Camera mode  = 4 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 4 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Execution ended after 0:00:01.229080747
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...

Honey_Patouceul · June 11, 2020, 8:56pm

I don’t have 2 cameras for trying but this seems working:

gst-launch-1.0 -ev nvarguscamerasrc ! 'video/x-raw(memory:NVMM), format=NV12, width=1280, height=720, framerate=30/1' ! tee name=t     t. ! queue ! comp.sink_0     t. ! queue ! comp.sink_1     nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=1280 sink_1::ypos=0  sink_0::width=1280 sink_0::height=720 sink_1::width=1280 sink_1::height=720 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! nvv4l2h264enc bitrate=8000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

And checked on same jetson with:

gst-launch-1.0 -ev udpsrc port=5000 ! queue ! application/x-rtp, media=video, encoding-name=H264 ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! videoconvert ! xvimagesink sync=false

Casmer · June 12, 2020, 6:46pm

Thank you so much! That helped me come up with a working pipeline. Now i need to receive it on a standard linux box.
I tried

gst-launch-1.0 -ev udpsrc port=5000 ! queue ! application/x-rtp, media=video, encoding-name=H264 , width=2560, height=720, framerate=30/1 ! rtph264depay ! h264parse !decodebin ! autovideosink

but received this and no video window:

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = application/x-rtp, media=(string)video, encoding-name=(string)H264, width=(int)2560, height=(int)720, framerate=(fraction)30/1, clock-rate=(int)90000
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, encoding-name=(string)H264, width=(int)2560, height=(int)720, framerate=(fraction)30/1, clock-rate=(int)90000
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/avdec_h264:avdec_h264-0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline, width=(int)2560, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01424028ffe1000a67424028965401400b7201000468ce3c80, level=(string)4, profile=(string)constrained-baseline
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/avdec_h264:avdec_h264-0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)2560, height=(int)720, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)0/1

foobar.warren · June 12, 2020, 9:01pm

I don’t know if this is applicable, but I’m using python, cv2 and numpy to ‘stack’ a webcam and pi camera.

If there is a gstreamer solution that would be more efficient however and I would switch over.

ret, frame = self.cam.cap.read()
ret, frame2 = self.picam.cap.read()
frameCombined = np.hstack((frame,frame2))