Audio Video sync within GStreamer

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

hi,

has anyone experienced sync failure between audio and video after ~30 seconds of encoding? it looks like the the omxh264enc stops synchronizing the audio and video in other words pulls incorrecty from the video queue and/or audio queue.

This is my gstreamer is composed of 3 pieplines, the pipelins were implemented in C but following is a line command which was converted manually (so it is possible that it won’t work as is):

v4l2src device = /dev/video0 ! intervideosink name="videoch1", format=3, enable-sync=false, allow-renegotiation=true 
alsasrc device="hw:tegrasndt186ref,0" ! tee ! queue max-size-time=3000000000, max-size-bytes=0, max-size-buffers=0, leaky=2 ! interaudiosink name="audioch1"
intervideosrc channel="videoch1" ! nvvidconv output-buffers=125 ! video/x-raw , width=(int)1280, height=(int)720, framerate=(fraction)60/1, format=(string)UYVY ! omxh264enc profile=8, bitrate=32000000, iframeinterval=15, control-rate=2 ! h264parse ! queue max-size-time=3000000000, max-size-bytes=0, max-size-buffers=0, leaky=2 ! mpegpsmux ! queue max-size-time=3000000000, max-size-bytes=0, max-size-buffers=0, leaky=2 ! filesink location=my.mpg
interaudiosrc channel="audioch1", format=3, enable-sync=false, allow-renegotiation=true ! video/x-raw , width=(int)1280, height=(int)720, framerate=(fraction)60/1, format=(string)UYVY ! voaacenc ! queue max-size-time=3000000000, max-size-bytes=0, max-size-buffers=0, leaky=2

Any suggestions will be appreciated.

2

Hi,
The synchronization is handled by gstreamer frameworks. Hardware encoder should be capable of outputting 720p60 H264 stream. Should be other factors triggering off synchronization.

It looks like you feed ‘video/x-raw,format=UYVY’ frames to encoder. UYVY is not supported and you should configure ‘video/x-raw(memory:NVMM),format=NV12’.

3

DaneLLL, hi,

thanks for the reply, I had problems with playing NV12 on PC after encoding I’ll give it another try (but I’ve added the UYVY to the driver a long time ago, so it works).
As part of the synchronization I’ve noticed that the CPU consumption is very high, I’ve tried to approach the gstreamer team but they pointed me back to you (NVidia).

How can I decrease the CPU utilization?

Discussion with gstreamer: [url]http://gstreamer-devel.966125.n4.nabble.com/InterVideoSrc-CPU-Utilization-td4691810.html[/url]

4

Hi,
The following pipeline is optimal on TX2:

$ gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=1920,height=1080,format=UYVY,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! omxh264enc ! fakesink

The reference camera is See3CAM CU135:

nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'UYVY'
        Name        : UYVY 4:2:2
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 3840x2160
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                Size: Discrete 4208x3120
                        Interval: Discrete 0.111s (9.000 fps)
                        Interval: Discrete 0.222s (4.500 fps)
                Size: Discrete 4096x2160
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.008s (120.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)

There is memocy copy in nvvidconv to copy ‘video/x-raw,format=UYVY’ buffers to video/x-‘raw(memory:NVMM),format=UYVY’ buffers, which consumes CPU cycles. So you may also see certain CPU consumption in your pipeline also.

We have it optimized in using tegra_multimedia_api. The sample is 12_camera_v4l2_cuda.

5

DaneLLL, hi,

your suggestion to replace I420 with NV12 actually dropped utilization by 50%. I had to remove the intervideosink and intervideosrc to accomplish this.

Though the main issue of audio synchronization is still there and it is more noticeable when the bitrate is high above 20Mbps. One can actually see that the lip-sync starts to diverge after 10-20 seconds, and then audio “ticks” are recorded.

Any suggestions?

6

Hi,
Since audio is full CPU loading path, you may execute ‘sudo jetson_clocks’ and try.
Also your source is ‘video:v4lsrc,audio:alsasrc’, you may try ‘video:videotestsrc,audio:alsasrc’ and ‘video:v4lsrc,audio:audiotestsrc’ for comparison.

7

Hi,

I have a problem with lip-sync, how the videotestsrc and audiotestsrc will support me on this one? it looks like the audio and video buffers drift apart that is the audio PTS (I guess) arrive much later than the video, though at the beginning they are at sync.

8

Hi,
AV off synchronization is mostly mismatch of timestamps in audio and video sources. It might help to compare ‘video:videotestsrc,audio:alsasrc’ and ‘video:v4lsrc,audio:audiotestsrc’. Since audiotestsrc and videotestsrc can be consistently outputted frames in steady interval. May help to recognize alsasrc or v4l2src is not stable.

Other users may share experience on debugging.

9

DaneLLL, hi,

we’ve noticed that the once the I420 was replaced by the NV12, though the CPU utilization was dropped, the frame rate was also dropped from 60 to 35. Could you please advise how fix this?

10

Hi,
Please apply the patch and try again. It looks not expected to have different result between NV12 and I420. Let’s run encoder at max clocks and check if helps the case.