Maximum AV sync possible with gstreamer 1.8.0 in L4T 24.2 at 23.976 FPS

We are doing a gstreamer application which requires AV sync of less than +/- 3 ms between audio and video. We use jetson TX1, gstreamer 1.8.0 and L4T24.2. Should support 23.976 FPS. We use Sync meter to measure the difference between an audio beep and a flash. The observations are at UHD 23.976 the sync meter reading is around +1 ms and remain constant sometimes. For other session the offset is observed to be around 43 ms which is equivalent to one video frame. We tried to print the audio and video timestamp using gst_pad_add_probe() and compare. But could not figure out the issue.Could we achieve this avsync tolerance with gstreamer. Is there another method to debug the issue.

Hi dawnpaul,
Could you share your gst command? What are your audio and video input devices?

If both devices gets timestamps in kernel, it would be good to print them out and make comparison in kernel.

Hi dawnpaul,
It can be usefule to print timestamp out in v4l2src/alsasrc. If we run the command:
$ gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=150 ! ‘video/x-raw,width=1280,height=960’ ! omxh264enc ! queue ! mux. alsasrc num-buffers=1000 do-timestamp=true ! voaacenc ! queue ! mux. qtmux name=mux ! filesink location=a.mp4

We can add prints in
gst-plugins-good-1.8.0/sys/v4l2/gstv4l2src.c
gst-plugins-base-1.8.0/ext/alsa/gstalsasrc.c

and rebuild the plugins. As timestamps are monotonic time, the comparison should give useful information.

We are using alsasink and nveglglessink. Application is Non-live video play-back. Got some replay from gstreamer forum.
http://gstreamer-devel.966125.n4.nabble.com/Maximum-AV-sync-possible-with-gstreamer-1-8-0-in-L4T-24-2-at-23-976-FPS-td4680593.html#a4680719
Still confused how to debug. Could we put any print in audio and video sink or driver and figureout where exactly is the problem. Offset is seen only once per 5 runs.

Hi dawnpaul,
Please try
1 Is the issue seen in max performance?
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/kernel/debug/clock/gbus/max > /sys/kernel/debug/clock/override.gbus/rate
echo 1 > /sys/kernel/debug/clock/override.gbus/state

2 Looks like it is h265 video stream. Is the issue seen with h264 stream?

1 Yes the issue is also seen in max performance

2 File is H265. Have not tested with H264

What calls my attention is that you say that this happens only once per 5 runs. Can you provide the file and pipeline that you are using?

What happens when you inspect the timestamps putting an identity element on the audio and video branch with the silent=false, in those cases where it doesn’t work do you see something weird on the timestamps?

We also have a tool called GstShark that helps to measure the time that each element holds a buffer but as Sebastian mentioned I am not too sure that is the problem because in theory the pipeline prerolls the data before playing. In any case you might find it interesting to see how the pipelines behaves:

https://developer.ridgerun.com/wiki/index.php?title=GstShark

Hi dawnpaul,
It is more like a issue in rendering. If it is an issue in h265 decoer, you should see video frames keep falling behind and get dropped. As it happens once in 5 runs, it looks like what Sebastian mentioned the first video frame possibly misses one VSYNC and gets displayed at next VSYNC.

For further analysis, we would like you to share the file, pipeline, and how to check if the issue happens.

Hi DaneLLL

We have some proprietary code included in demux and sink elements. Sharing a command which uses the elements that are available in the L4T package. Replaced nveglglessink with nvoverlaysink.

gst-launch-1.0 filesrc location=outputpcmaac.mp4 ! qtdemux name=demux demux.audio_0 ! queue ! avdec_aac ! audioconvert ! audio/x-raw , layout=interleaved, rate=48000, format=S16LE, channels=2 ! alsasink device= hw:Tegra,3 demux.video_0 ! queue max-size-buffers=500 max-size-bytes=16800000 max-size-time=2000000000 ! h265parse ! omxh265dec ! nvoverlaysink name=app -e

Here is the link to the video file.


File is of resolution 3840x2160 and frame rate 23.976

AV sync offset observations:
average offset: 15.9 ms (average offset for a total of 10 test runs)
span = 32 ms (difference between highest offset and lowest offset from 10 test runs)

Hi dawnpaul,
Please also share how you profile the AV sync offset.

we’re using the equipment from sync-one2.co.uk

Hi DaneLLL,
Would you require more info from us to test this at your end?

Hi dawnpaul,
We are still looking for a way to detect the one frame delay as it is impossible to physically tell it.

Looks like you are using the 3rd party equipment to do the check but unfortunately we don’t have the device.

Hi dawnpaul,
Unfortunately we are not able to help on this further. After internal discussion, we think 1-frame delay is within tolerance and it looks to be race condition that the video frame just misses the VSYNC and gets displayed at next VSYNC.

Wish other users can share their experience. Thanks.