using gstreamer to transcode/downscale 1080i videos on Jetson TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I have an HD-PVR which outputs a .ts stream (H.264 AVCHD video, and AC-3 audio) in whatever resolution is connected to the input. In this case the feed is a 1080i video stream, and I pipe it to a file for subsequent transcoding.

I like gstreamer, because it is quick. However, it can be a pig at times. I have included the pipeline below.

The problem is that, in the resultant mp4 file, the video is only half the duration (even though the audio is the full 1 minute long). I have included the mediainfo results for both the source and sink files.
Playback is choppy, pixellated, and at twice the speed (with pauses).

I have seen in other threads that Tegra has issues with generating interlaced video. However, I am generating progressive formats (720p), so I am at a loss to understand the cause of this issue.

Does anyone have any helpful insights?

gst-launch-1.0 -v -e filesrc location=/home/nvidia/input.ts typefind=true
! decodebin name=demux
demux.
! queue
! audio/x-raw,format=S16LE,rate=48000
! mux.audio_0
demux.
! queue
! nvvidconv
! video/x-raw,format=I420,width=1280,height=720,framerate=30/1
! x264enc bframes=0 b-adapt=false speed-preset=1 tune=0x00000004
! mux.video_0
qtmux name=mux
! filesink sync=true location=/home/nvidia/output.mp4

SOURCE FILE mediainfo :

General
ID : 1 (0x1)
Complete name : input.ts
Format : MPEG-TS
File size : 53.5 MiB
Duration : 1mn 1s
Overall bit rate mode : Variable
Overall bit rate : 7 202 Kbps

Video
ID : 256 (0x100)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L4
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Format settings, GOP : M=4, N=32
Codec ID : 27
Duration : 1mn 2s
Bit rate mode : Variable
Maximum bit rate : 20.0 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 30.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced #####
Scan type, store method : Separated fields
Scan order : Top Field First
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

Audio
ID : 257 (0x101)
Menu ID : 1 (0x1)
Format : AAC
Format/Info : Advanced Audio Codec
Format version : Version 2
Format profile : LC
Muxing mode : ADTS
Codec ID : 15
Duration : 1mn 1s
Bit rate mode : Variable
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Frame rate : 46.875 fps (1024 spf)
Compression mode : Lossy
Delay relative to video : -100ms

Menu
ID : 4096 (0x1000)
Menu ID : 1 (0x1)
Duration : 1mn 1s
List : 256 (0x100) (AVC) / 257 (0x101) (AAC)
Service name : Service01
Service provider : FFmpeg
Service type : digital television

SINK FILE mediainfo :

General
Complete name : output.mp4
Format : MPEG-4
Format profile : QuickTime
Codec ID : qt 2005.03 (qt )
File size : 41.0 MiB
Duration : 1mn 2s
Overall bit rate : 5 539 Kbps
Encoded date : UTC 2018-03-15 20:35:15
Tagged date : UTC 2018-03-15 20:35:15

Video
ID : 2
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Baseline@L3.1
Format settings, CABAC : No
Format settings, ReFrames : 1 frame
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 30s 100ms ##### HALF THE SOURCE DURATION
Bit rate : 2 048 Kbps
Maximum bit rate : 2 097 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 30.000 fps ##### SAME FRAMERATE
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.074
Stream size : 29.5 MiB (72%)
Writing library : x264 core 148 r2643 5c65704
Encoding settings : cabac=0 / ref=1 / deblock=0:0:0 / analyse=0:0 / me=dia / subme=0 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0 / threads=4 / lookahead_threads=4 / sliced_threads=1 / slices=4 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=300 / keyint_min=30 / scenecut=0 / intra_refresh=0 / rc_lookahead=0 / rc=cbr / mbtree=0 / bitrate=2048 ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=2048 / vbv_bufsize=1228 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=0
Language : English
Encoded date : UTC 2018-03-15 20:35:15
Tagged date : UTC 2018-03-15 20:35:15

Audio
ID : 1
Format : PCM
Format settings, Endianness : Little
Format settings, Sign : Signed
Codec ID : sowt
Duration : 1mn 2s
Bit rate mode : Constant
Bit rate : 1 536 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Stream size : 11.4 MiB (28%)
Language : English
Encoded date : UTC 2018-03-15 20:35:15
Tagged date : UTC 2018-03-15 20:35:15

2

Hi hilndr44,
For interlaced h264 stream input, it will be outputted as progressive YUVs. Do you see issues with the progressive YUV outputs?

3

In the “SINK FILE mediainfo” I listed, you can see that the final output is progressive YUV at 30 fps, but the video duration is only half that of the input. And, as I have stated, the playback is choppy and pixellated, with double-speed playback, interrupted by long pauses. So, yes, I do see issues.

Input :
Duration : 1mn 2s

Output :
Duration : 30s 100ms

4

Looks like an issue in timestamp. Have you tried ffdec_h264 or avdec_h264?

5

Hi DaneLLL, thank you for the responses.

I currently do not have either of those plug-ins. Which package would I need to install?

6

gstreamer1.0-libav

gst-inspect-1.0 avdec_h264

Factory Details:
Rank primary (256)
Long-name libav H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 decoder
Klass Codec/Decoder/Video
Description libav h264 decoder
Author Wim Taymans wim.taymans@gmail.com, Ronald Bultje rbultje@ronald.bitfreak.net, Edward Hervey bilboed@bilboed.com

Plugin Details:
Name libav
Description All libav codecs and formats (system install)
Filename /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so
Version 1.8.3
License LGPL
Source module gst-libav
Binary package libav
Origin URL http://www.libav.org

7

Thanks.

I tried avdec_h264 in the pipline, but the results are similar.
Duration is still cut in half (from 1 minute, down to 30 seconds), and resultant framerate is 62.06 fps.

Looking at the pipeline’s verbose output, it seems to think it is doing 30 fps throughout the pipeline, but that is not what is in the output mp4 file. I don’t know how it got 62.06 fps; am I missing some kind of “do-timestamp” in the pipe? I tried it once, but I don’t think it has any effect for non-live video.

I also suspect that it is doing some kind of deinterlace which combines 2 frames into one. Hence, we only get half the video duration (which gets mux’d with the full-length audio duration).

Here is the updated pipeline (note that I removed the 1080->720 scaling so that we’ll have one less variable while debugging, and I broke-out decodebin into its constituent parts so that I could swap-in avdec_h264) :

#!/bin//sh

BASE_NAME=$(basename -s .ts $1)
echo $BASE_NAME

gst-launch-1.0 -v -e filesrc location=/mnt/evo250gb/videos_to_be_converted/$1
! tsdemux name=demux
! queue
! video/x-h264, stream-format=byte-stream, alignment=nal, interlace-mode=interleaved
! h264parse
! avdec_h264
! nvvidconv
! omxh264enc
! mux.video_0
demux.
! queue
! audio/mpeg, mpegversion=2, stream-format=adts
! aacparse
! audio/mpeg
! faad
! audio/x-raw, format=S16LE, layout=interleaved, rate=48000, channels=2
! qtmux name=mux
! filesink sync=false location=/mnt/evo250gb/video_output/$BASE_NAME.mp4

Here is the verbose output :

Setting pipeline to PAUSED …
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingInside NvxLiteH265DecoderLowLatencyInitNvxLiteH265DecoderLowLatencyInit set DPB and MjstreamingPipeline is PREROLLING …
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = “video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)nal”
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = “video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)nal”
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = “video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)nal”
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = “audio/mpeg,\ mpegversion=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = “audio/mpeg,\ mpegversion=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = “audio/mpeg,\ mpegversion=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstAacParse:aacparse0.GstPad:sink: caps = “audio/mpeg,\ mpegversion=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = “audio/mpeg,\ mpegversion=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = “video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)nal,\ interlace-mode=(string)interleaved”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = “video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)nal”
/GstPipeline:pipeline0/GstAacParse:aacparse0.GstPad:src: caps = “audio/mpeg,\ framed=(boolean)true,\ mpegversion=(int)2,\ level=(string)2,\ profile=(string)lc,\ rate=(int)48000,\ channels=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = “audio/mpeg,\ framed=(boolean)true,\ mpegversion=(int)2,\ level=(string)2,\ profile=(string)lc,\ rate=(int)48000,\ channels=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstFaad:faad0.GstPad:sink: caps = “audio/mpeg,\ framed=(boolean)true,\ mpegversion=(int)2,\ level=(string)2,\ profile=(string)lc,\ rate=(int)48000,\ channels=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = “audio/mpeg,\ framed=(boolean)true,\ mpegversion=(int)2,\ level=(string)2,\ profile=(string)lc,\ rate=(int)48000,\ channels=(int)2,\ stream-format=(string)adts”
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = “video/x-h264,\ stream-format=(string)avc,\ alignment=(string)au,\ interlace-mode=(string)interleaved,\ pixel-aspect-ratio=(fraction)1/1,\ width=(int)1920,\ height=(int)1080,\ framerate=(fraction)30/1,\ parsed=(boolean)true,\ profile=(string)main,\ level=(string)4,\ codec_data=(buffer)014d4028ffe10025274d40289a6280f0088fbc07d404040500000303e80000ea60e8c0004c4b0002faf2ef380a01000428fe3c80”
/GstPipeline:pipeline0/GstFaad:faad0.GstPad:src: caps = “audio/x-raw,\ format=(string)S16LE,\ layout=(string)interleaved,\ rate=(int)48000,\ channels=(int)2,\ channel-mask=(bitmask)0x0000000000000003”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:src: caps = “audio/x-raw,\ format=(string)S16LE,\ layout=(string)interleaved,\ rate=(int)48000,\ channels=(int)2,\ channel-mask=(bitmask)0x0000000000000003”
/GstPipeline:pipeline0/GstQTMux:mux.GstPad:audio_0: caps = “audio/x-raw,\ format=(string)S16LE,\ layout=(string)interleaved,\ rate=(int)48000,\ channels=(int)2,\ channel-mask=(bitmask)0x0000000000000003”
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:sink: caps = “audio/x-raw,\ format=(string)S16LE,\ layout=(string)interleaved,\ rate=(int)48000,\ channels=(int)2,\ channel-mask=(bitmask)0x0000000000000003”
Redistribute latency…
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:sink: caps = “video/x-h264,\ stream-format=(string)avc,\ alignment=(string)au,\ interlace-mode=(string)interleaved,\ pixel-aspect-ratio=(fraction)1/1,\ width=(int)1920,\ height=(int)1080,\ framerate=(fraction)30/1,\ parsed=(boolean)true,\ profile=(string)main,\ level=(string)4,\ codec_data=(buffer)014d4028ffe10025274d40289a6280f0088fbc07d404040500000303e80000ea60e8c0004c4b0002faf2ef380a01000428fe3c80”
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:src: caps = “video/x-raw,\ format=(string)I420,\ width=(int)1920,\ height=(int)1080,\ interlace-mode=(string)mixed,\ pixel-aspect-ratio=(fraction)1/1,\ chroma-site=(string)mpeg2,\ colorimetry=(string)bt709,\ framerate=(fraction)30/1”
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = “video/x-raw(memory:NVMM),\ width=(int)1920,\ height=(int)1080,\ format=(string)I420,\ framerate=(fraction)30/1,\ interlace-mode=(string)mixed,\ pixel-aspect-ratio=(fraction)1/1”
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:sink: caps = “video/x-raw(memory:NVMM),\ width=(int)1920,\ height=(int)1080,\ format=(string)I420,\ framerate=(fraction)30/1,\ interlace-mode=(string)mixed,\ pixel-aspect-ratio=(fraction)1/1”
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = “video/x-raw,\ format=(string)I420,\ width=(int)1920,\ height=(int)1080,\ interlace-mode=(string)mixed,\ pixel-aspect-ratio=(fraction)1/1,\ chroma-site=(string)mpeg2,\ colorimetry=(string)bt709,\ framerate=(fraction)30/1”
Redistribute latency…
===== MSENC blits (mode: 1) into tiled surfaces =====
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:src: caps = “video/x-h264,\ alignment=(string)au,\ stream-format=(string)avc,\ width=(int)1920,\ height=(int)1080,\ pixel-aspect-ratio=(fraction)1/1,\ framerate=(fraction)30/1”
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:src: caps = “video/x-h264,\ alignment=(string)au,\ stream-format=(string)avc,\ width=(int)1920,\ height=(int)1080,\ pixel-aspect-ratio=(fraction)1/1,\ framerate=(fraction)30/1,\ codec_data=(buffer)014240150301000b6742403495a01e0089f95001000468ce3c80”
/GstPipeline:pipeline0/GstQTMux:mux.GstPad:video_0: caps = “video/x-h264,\ alignment=(string)au,\ stream-format=(string)avc,\ width=(int)1920,\ height=(int)1080,\ pixel-aspect-ratio=(fraction)1/1,\ framerate=(fraction)30/1,\ codec_data=(buffer)014240150301000b6742403495a01e0089f95001000468ce3c80”
/GstPipeline:pipeline0/GstQTMux:mux.GstPad:src: caps = “video/quicktime,\ variant=(string)apple”
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = “video/quicktime,\ variant=(string)apple”
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock

8

Also, I noticed that if I use x264enc, I get 30 fps in the output mp4 file. When I use omxh264enc, the framerate is either 62.06 fps or 124.12 fps.

So, as I see it, I have two issues :

  1. omxh264enc produces the wrong framerate

  2. somewhere in the pipeline, a virtual deinterlacing(?) process is cutting the video duration in half

9

Please share input.ts so that we can reproduce the issue.

10

The video is in this dropbox.

https://www.dropbox.com/l/scl/AAA_oIa8e8SZAkC_9dKEsZ5BO-VOQNYBgjk

11

Actual filename is : 2018_03_12_15_56_Test_Video_1_minutes.ts

12

Hi hilndr44,

Your dropbox download have request access, just send the message to you, please approve. Thanks!

13

access is approved

14

Hi hilndr44,
Is is an issue in tsdemux which does not give correct timestamp. you can try a full 3rdparty pipeline:

$ export DISPLAY=:0
$ gst-launch-1.0 filesrc location= 2018_03_12_15_56_Test_Video_1_Minutes.ts ! tsdemux ! h264parse ! avdec_h264 ! xvimagesink

The video is also played in 60fps.

The elements omxh264enc/omxh264dec do not overwrite timestamp. The timestamps are simply passed from input buffers to output buffers.

15

Hi DaneLLL,

That is too bad about tsdemux. The .ts video plays fine (at 30 fps) in VLC on a windows PC.

I am also able to convert/downscale it on the Jetson using ffmpeg (but it takes 4x the amount of time as gstreamer) :

ffmpeg -i 2018_03_12_15_56_Test_Video_1_Minutes.ts -filter:v scale=1280:-1 -acodec copy -bsf:a aac_adtstoasc 2018_03_12_15_56_Test_Video_1_Minutes.mp4

The resultant mp4 video (also on the dropbox) plays fine in VLC, too.

Do you know of any other demux that would work in my situation?

16

Hi hilndr44,
Please try to get suggestion from http://gstreamer-devel.966125.n4.nabble.com/

17

Just some follow-up information.

The folks on nabble suggested that I run with a more up-to-date version : gstreamer-1.14
I upgraded the Jetson TX2 to the latest jetpack 3.2, but it only has gstreamer-1.8

For now, I am using ffmpeg, but it does not make use of the GPU so, it is very slow.

18

Hi hhilndr44,
You can refer to [GSTREAMER BUILD INSTRUCTIONS] in
https://developer.nvidia.com/embedded/dlc/l4t-accelerated-gstreamer-guide-28-2-ga
and manually update to 1.12.4

19

Hi hilndr44,
Reference this URL, https://devtalk.nvidia.com/default/topic/1002969
ffmpeg doesn’t support yet.