Composite video capture with gst-launch-1.0 combined with GoPro A/V feed for QGroundControl FPV

Robotics & Edge Computing Jetson & Embedded Systems Jetson TK1
1

I am trying to use an old analog capture device to view my UAV udp video feed in qgroundcontrol but I can not seem to get the gst-launch params correct on TK1.

SOLVED:
Here is the gst-launch command for analog video capture at 30/1 30000/1001 framerate:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-timestamp=true norm=NTSC pixel-aspect-ratio=1 ! video/x-raw, format=YUY2, width=720, height=480 ! queue max-size-buffers=0 max-size-time=0 max-size-bytes=0 ! tee name=t ! queue ! videoconvert ! omxh264enc ! video/x-h264, width=720, height=480 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=192.168.1.79 port=5600 t. ! queue ! videoconvert

Here is the setup:

Using cx231xx capture device to capture analog gopro feed through composite cables and gopro usb-to-composite: OTG102 - LinuxTVWiki and https://www.ebay.com/itm/Gopro-Hero3-USB-to-AV-Video-Output-RCA-Combo-Cable-5V-DC-Power-Supply-Cable-FPV/261167355923

OTG2 device captures video fine through VLC on Jetson-TK1 selecting Analog-TV and NTSC.

Here is my gst-launch-1.0 command that I thought would work following https://www.linuxtv.org/wiki/index.php/V4L_capturing:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-timestamp=true norm=NTSC pixel-aspect-ratio=1 ! tee name=t ! queue ! videoconvert ! omxh264enc ! video/x-h264, width=720, height=480, stream-format=byte-stream ! h264parse ! rtph264pay config-interval=1 ! udpsink host=192.168.1.79 port=5600 t. ! queue ! videoconvert

gst-launch-1.0 command that works with generic v4l2 webcam:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 ! tee name=t ! queue ! videoconvert ! omxh264enc ! video/x-h264, width=1920, height=1080, framerate=30/1, stream-format=byte-stream ! h264parse ! rtph264pay config-interval=1 ! udpsink host=192.168.1.79 port=5600 t. ! queue ! videoconvert

Can anyone here chime in and help out with the correct params I should be using? Thanks in advance for any pointers…

Here are the capabilities of the cx231xx device:

gst-launch-1.0 --gst-debug=v4l2src:5 v4l2src device=/dev/video0 ! fakesink 2>&1 | sed -une '/caps of src/ s/[:;] /\n/gp'
0:00:00.155975694  6598   0x17ee30 DEBUG                v4l2src gstv4l2src.c:300:gst_v4l2src_negotiate:<v4l2src0> caps of src
video/x-raw, format=(string)YUY2, framerate=(fraction)30000/1001, width=(int)[ 48, 720 ], height=(int)[ 32, 480 ], interlace-mode=(string)interleaved, pixel-aspect-ratio=(fraction)54/59, colorimetry=(string)bt601
video/x-raw, format=(string)I420, framerate=(fraction)30000/1001, width=(int)[ 48, 720 ], height=(int)[ 32, 480 ], interlace-mode=(string)interleaved, pixel-aspect-ratio=(fraction)54/59, colorimetry=(string)bt601
video/x-raw, format=(string)YV12, framerate=(fraction)30000/1001, width=(int)[ 48, 720 ], height=(int)[ 32, 480 ], interlace-mode=(string)interleaved, pixel-aspect-ratio=(fraction)54/59, colorimetry=(string)bt601
video/x-raw, format=(string)BGR, framerate=(fraction)30000/1001, width=(int)[ 48, 720 ], height=(int)[ 32, 480 ], interlace-mode=(string)interleaved, pixel-aspect-ratio=(fraction)54/59, colorimetry=(string)1:1:5:4
video/x-raw, format=(string)RGB, framerate=(fraction)30000/1001, width=(int)[ 48, 720 ], height=(int)[ 32, 480 ], interlace-mode=(string)interleaved, pixel-aspect-ratio=(fraction)54/59, colorimetry=(string)1:1:5:4
2

Not tried on TK1, but on TX1/TX2, I need to add baseline profile into caps after omxh264enc for such UDP streaming case.

video/x-h264, profile=baseline , ...

.
Be also aware that default bitrate of encoder is 4M, you may increase if you have high resolution or framerate. Some details here.

3

Great thanks for the input ill give the baseline a try ;)

4

Gave that a try and still no video in xvimagesink test. Im sure with enough fiddling with params i can get this just didn’t think it was that much different then a v4l2 webcam source…

New command here still no playback:

gst-launch-1.0 -e v4l2src device=/dev/video0 norm=NTSC ! video/x-raw, format=YUY2, width=720, height=480 ! tee name=t_vid ! queue ! videoconvert ! omxh264enc ! video/x-h264, profile=baseline ! h264parse ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5600 t_vid. ! queue ! videoconvert

Command to view on tk1:

gst-launch-1.0 udpsrc port=5600 ! application/x-rtp, encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! avdec_h264 ! xvimagesink
5

Okay with the following command I get more debug output but still no video:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-timestamp=true norm=NTSC pixel-aspect-ratio=1 ! tee name=t ! queue ! videoconvert ! omxh264enc ! video/x-h264, profile=baseline, width=720, height=480, stream-format=byte-stream ! h264parse ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5600 t. ! queue ! videoconvert

Output:

Setting pipeline to PAUSED ...
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingPipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstTee:t.GstTeePad:src_0: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstTee:t.GstTeePad:src_1: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
Framerate set to : 30 at NvxVideoEncoderSetParameter/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstTee:t.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
NvMMLiteOpen : Block : BlockType = 4
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)30000/1001\,\ width\=\(int\)720\,\ height\=\(int\)480\,\ interlace-mode\=\(string\)interleaved\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt601"
6

Sorry I missed that, but it may be because of the second subpipe after tee having no sink (ends with videoconvert).
Can you try adding fakesink at the end (or remove tee and the second subpipe) ?

7

Gonna give that a try ill report back in a few… Thanks for the input my friend :)

8

Well this is odd when forcing NTSC and framerate to framerate=30000/1001 I get this in cx231xx dmesg:

cx25840 6-0044: 720x480 is not a valid size!

Yet v4l2-ctl reports 720x480 as the correct settings for NTSC 24.94 fps. Im still tinkering with the commands its gotta be something simple since vlc just plays 720x480 fine.

I did notice that the device defaults to PAL and in the kernel logs after success in vlc i see this:

[  486.805301] cx231xx #0: do_mode_ctrl_overrides : 0xff
[  486.806239] cx231xx #0: do_mode_ctrl_overrides PAL
[  486.846339] cx231xx #0: do_mode_ctrl_overrides : 0xb000
[  486.847237] cx231xx #0: do_mode_ctrl_overrides NTSC

So maybe vlc is somehow setting the override to NTSC that gst-launch can’t…

I looked in /sys/module/cx231xx/params and didnt see a module override but im still digging…

do_mode_ctrl_overrides=0xb000

9

Some more cx231xx kernel logs here:

Main errors are

UsbInterface::sendCommand, failed with status --71

and

cx231xx #0: can't change interface 3 alt no. to 0 (err=-71)

[   15.942345] cx25840 6-0044: loaded v4l-cx231xx-avcore-01.fw firmware (16382 bytes)
[   15.985308] cx231xx #0: cx231xx #0: v4l2 driver version 0.0.2
[   16.004880] cx231xx #0: cx231xx_dif_set_standard: setStandard to ffffffff
[   16.072277] cx231xx #0: video_mux : 0
[   16.072289] cx231xx #0: do_mode_ctrl_overrides : 0xff
[   16.073828] cx231xx #0: do_mode_ctrl_overrides PAL
[   16.132289] cx231xx #0: cx231xx #0/0: registered device video1 [v4l2]
[   16.132407] cx231xx #0: cx231xx #0/0: registered device vbi0
[   16.132413] cx231xx #0: V4L2 device registered as video1 and vbi0
[   16.132423] cx231xx #0: EndPoint Addr 0x84, Alternate settings: 5
[   16.132428] cx231xx #0: Alternate setting 0, max size= 512
[   16.132434] cx231xx #0: Alternate setting 1, max size= 184
[   16.132440] cx231xx #0: Alternate setting 2, max size= 728
[   16.132444] cx231xx #0: Alternate setting 3, max size= 2892
[   16.132449] cx231xx #0: Alternate setting 4, max size= 1800
[   16.132455] cx231xx #0: EndPoint Addr 0x85, Alternate settings: 2
[   16.132460] cx231xx #0: Alternate setting 0, max size= 512
[   16.132465] cx231xx #0: Alternate setting 1, max size= 512
[   16.132470] cx231xx #0: EndPoint Addr 0x86, Alternate settings: 2
[   16.132475] cx231xx #0: Alternate setting 0, max size= 512
[   16.132480] cx231xx #0: Alternate setting 1, max size= 576
[   16.132733] usbcore: registered new interface driver cx231xx
[   16.152280] cx231xx #0: cx231xx_stop_stream():: ep_mask = 8
[   16.163853] cx231xx #0:  setPowerMode::mode = 48, No Change req.
[   16.166694] cx231xx #0: cx231xx_stop_stream():: ep_mask = 8
[   16.175815] cx231xx #0: cx231xx-audio.c: probing for cx231xx non standard usbaudio
[   16.176245] cx231xx #0: EndPoint Addr 0x83, Alternate settings: 3
[   16.176249] cx231xx #0: Alternate setting 0, max size= 512
[   16.176252] cx231xx #0: Alternate setting 1, max size= 28
[   16.176254] cx231xx #0: Alternate setting 2, max size= 52
[   16.176259] cx231xx: Cx231xx Audio Extension initialized
[   16.456834] tegradc tegradc.1: nominal-pclk:148500000 parent:594000000 div:4.0 pclk:148500000 147015000~161865000
[   16.457053] tegra_dc_hdmi_enable: HDMI clock already configured to target frequency, skipping clk setup.
[   21.608507] cx231xx #0: cx231xx_initialize_stream_xfer: Audio enter HANC
[   21.608859] cx231xx #0: cx231xx_start_stream():: ep_mask = 4
[   21.609485] cx231xx #0: cx231xx_stop_stream():: ep_mask = 4
[   21.610589] cx231xx #0: cx231xx_initialize_stream_xfer: Audio enter HANC
[   21.610921] cx231xx #0: cx231xx_start_stream():: ep_mask = 4
[   21.611271] cx231xx #0: cx231xx_stop_stream():: ep_mask = 4
[   21.612135] cx231xx #0: cx231xx_initialize_stream_xfer: Audio enter HANC
[   21.612462] cx231xx #0: cx231xx_start_stream():: ep_mask = 4
[   21.612966] cx231xx #0: cx231xx_stop_stream():: ep_mask = 4
[   21.614053] cx231xx #0: cx231xx_initialize_stream_xfer: Audio enter HANC
[   21.614309] cx231xx #0: cx231xx_start_stream():: ep_mask = 4
[   21.615065] cx231xx #0: cx231xx_stop_stream():: ep_mask = 4
[   21.616470] cx231xx #0: cx231xx_initialize_stream_xfer: Audio enter HANC
[   21.616809] cx231xx #0: cx231xx_start_stream():: ep_mask = 4
[   21.626215] cx231xx #0: cx231xx_stop_stream():: ep_mask = 4
[   21.644796] cx231xx #0: cx231xx_initialize_stream_xfer: Audio enter HANC
[   21.645119] cx231xx #0: cx231xx_start_stream():: ep_mask = 4
[   21.654654] cx231xx #0: cx231xx_init_audio_isoc: Starting ISO AUDIO transfers
[   22.660515] warning: process `colord-sane' used the deprecated sysctl system call with 8.1.2.
[   27.566509] cx231xx #0: cx231xx_stop_stream():: ep_mask = 4
[   38.802083] r8169 0000:03:00.0 eth0: link up
[   38.802863] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  161.120784] nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead.
[  209.547338] cx231xx #0:  setPowerMode::mode = 48, No Change req.
[  209.582333] cx231xx #0: do_mode_ctrl_overrides : 0xb000
[  209.583735] cx231xx #0: do_mode_ctrl_overrides NTSC
[  209.644266] cx231xx #0: cx231xx_stop_stream():: ep_mask = 8
[  209.649324] cx231xx #0: cx231xx_initialize_stream_xfer: set video registers
[  209.649676] cx231xx #0: cx231xx_start_stream():: ep_mask = 8
[  212.755189] [sched_delayed] sched: RT throttling activated
[  215.218585] cx231xx #0: UsbInterface::sendCommand, failed with status --71
[  215.223560] cx231xx #0: UsbInterface::sendCommand, failed with status --71
[  215.249613] cx231xx #0: UsbInterface::sendCommand, failed with status --71
[  215.254100] cx231xx #0: UsbInterface::sendCommand, failed with status --71
[  215.289493] cx231xx #0: UsbInterface::sendCommand, failed with status --71
[  215.293607] cx231xx #0: UsbInterface::sendCommand, failed with status --71
[  215.367425] cx231xx #0: cx231xx_stop_stream():: ep_mask = 8
[  215.380643] cx231xx #0: can't change interface 3 alt no. to 0 (err=-71)

This looks promising: [PATCH] Fix regression introduced which broke the Hauppauge USBLive 2

10

Some success here. I’ve gotten the feed to display locally with the following command:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-timestamp=true norm=NTSC pixel-aspect-ratio=1 ! video/x-raw, format=YUY2, width=720, height=480 ! queue max-size-buffers=0 max-size-time=0 max-size-bytes=0 ! xvimagesink

Now all I need is to add the correct rtph264pay ! udpsink ! and videoconvert params (if needed) not sure if I can even use omxh264enc with analog video/x-raw.

Getting somewhere at least ;)

11

Got it (somewhat). This command does the trick for rdp streaming analog NTSC BUT if the stream is stopped and restarted the system will lock up (at least in my case) and the patch above may be the cause or solution.

Analog NTSC UDP Streaming command

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-timestamp=true norm=NTSC pixel-aspect-ratio=1 ! video/x-raw, format=YUY2, width=720, height=480 ! queue max-size-buffers=0 max-size-time=0 max-size-bytes=0 ! tee name=t ! queue ! videoconvert ! omxh264enc ! video/x-h264, width=720, height=480 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=192.168.1.79 port=5600 t. ! queue ! videoconvert

Lockup kernel message:

cx231xx #0: can't change interface 3 alt no. to 0 (err=-71)

Google it ;)

12

Well this lock-up is a major issue for a UAV since the cx231xx driver is working but locks up system and in flight this is obviously unwanted so I need to figure out what is causing this lockup or move to a 2 camera system with a webcam (= more weight).

13

Scratch that lockup issue and this is TOTALLY my fault. I had my TX1 19v power supply connected to my TK1 that has a max input of 12v and hence lots of lockups and thankfully it is okay. I tested the same gst-launch command on my actual Toradex Apalis TK1 Drone with GoPro attached and all is well and smooth. I do most testing on tk1 1st and this is my fault for not realizing i had the wrong power supply connected.

14

Just to clarify the gst-launch command the videoconvert ending pipline is from jetson redtail gscam here:
[url]redtail/caffe_gscam_h264.launch at master · NVIDIA-AI-IOT/redtail · GitHub

And here is H265:
[url]redtail/caffe_gscam_h265.launch at master · NVIDIA-AI-IOT/redtail · GitHub

Framerate is what v4l2-ctl -d /dev/video0 reports for my devices capture framerate. v4l2-clt is very helpful in reporting your devices hardware abilities.