Video drops frames/AV desync

[paulgheorghe94]

Hello,

I have a project for the Jetson Nano where I receive AV from HDMI and I try to detect scene changes from audio and video and the length between the scene changes. For this, I need the audio and video to be perfectly in sync (for audio I detect silence and for video I detect dramatic color change).

For this I have 2 pipelines in gstreamer 1.14.5:

pipeline_video_str = ‘v4l2src device=/dev/video0 do-timestamp=true blocksize=64000 ! queue !
image/jpeg,width=1280,height=720,framerate=30/1 ! videorate ! queue !
appsink name=videosink alsasrc device=hw:CARD=MS2109,DEV=0 blocksize=6400 do-timestamp=true ! queue !
audioconvert ! audioresample !
audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! appsink name=audiosink’

This one receives video and audio and sends it to Python callbacks, where I perform the analysis. Then, I forward audio and video to 2 appsrc, to display and play them:

pipeline2_str = ‘appsrc name=videosrc do-timestamp=true format=3 ! jpegparse ! jpegdec ! queue !
nvoverlaysink sync=false
appsrc name=audiosrc format=3 do-timestamp=true ! audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! audioconvert !
queue ! alsasink device=hw:0,3 sync=false’

This works fine for a bit,

but after a while 2 things start to happen, which may be strongly linked:
Frames seem to start dropping and AV becomes desynchronized. I notice these problems because I assume I have 30 FPS and, for example, if I detect a 30 seconds length part, I have received less than 900 frames (something like 877-889 frames). The desync is apparent because the scene change detector becomes inaccurate (so audio silence does not line up with color change). I want to mention that the CPU is not maxed out, I perform computation on GPU as well.

I would appreciate help regarding:

1. How can I better debug my setup? I tried printing the “drop” property of “videorate”, but it does not seem consistent with what I am experiencing. I want to make sure that my bug happens because of dropped frames.
2. Can I improve the pipelines to not drop frames? I tried adding “max-size-buffers=0 max-size-bytes=0 max-size-time=0” to queues, but I still experience issues.
3. What is the best approach to keep the synchronization?

Thank you very much for reading, this forum was always helpful to me, it’s great!

[DaneLLL]

Hi,
You may remove sync=false and try. There is synchronization mechanism in gstreamer and sync=false disables it. Please enable it for a try.

And please execute sudo nvpmodel -m 0 and sudo jetson_clocks, to run system at maximum performance. For checking if there is bottleneck in CPU or other hardware blocks, please execute sudo tegrastats to get system status.

[paulgheorghe94]

Hello,

Thank you for the response. I forgot to mention, sync=true makes the video have 1 FPS and no audio. I tried adding PTS and DTS to the buffers, but no luck so far.

[DaneLLL]

Hi,
You may try gst-launch-1.0 command and check if audio and video are synchronized. OpenCV takes significant CPU usage and it may have impact to performance.