Video drops frames/AV desync


I have a project for the Jetson Nano where I receive AV from HDMI and I try to detect scene changes from audio and video and the length between the scene changes. For this, I need the audio and video to be perfectly in sync (for audio I detect silence and for video I detect dramatic color change).

For this I have 2 pipelines in gstreamer 1.14.5:

pipeline_video_str = ‘v4l2src device=/dev/video0 do-timestamp=true blocksize=64000 ! queue !
image/jpeg,width=1280,height=720,framerate=30/1 ! videorate ! queue !
appsink name=videosink alsasrc device=hw:CARD=MS2109,DEV=0 blocksize=6400 do-timestamp=true ! queue !
audioconvert ! audioresample !
audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! appsink name=audiosink’

This one receives video and audio and sends it to Python callbacks, where I perform the analysis. Then, I forward audio and video to 2 appsrc, to display and play them:

pipeline2_str = ‘appsrc name=videosrc do-timestamp=true format=3 ! jpegparse ! jpegdec ! queue !
nvoverlaysink sync=false
appsrc name=audiosrc format=3 do-timestamp=true ! audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! audioconvert !
queue ! alsasink device=hw:0,3 sync=false’

This works fine for a bit,

but after a while 2 things start to happen, which may be strongly linked:
Frames seem to start dropping and AV becomes desynchronized. I notice these problems because I assume I have 30 FPS and, for example, if I detect a 30 seconds length part, I have received less than 900 frames (something like 877-889 frames). The desync is apparent because the scene change detector becomes inaccurate (so audio silence does not line up with color change). I want to mention that the CPU is not maxed out, I perform computation on GPU as well.

I would appreciate help regarding:

  1. How can I better debug my setup? I tried printing the “drop” property of “videorate”, but it does not seem consistent with what I am experiencing. I want to make sure that my bug happens because of dropped frames.
  2. Can I improve the pipelines to not drop frames? I tried adding “max-size-buffers=0 max-size-bytes=0 max-size-time=0” to queues, but I still experience issues.
  3. What is the best approach to keep the synchronization?

Thank you very much for reading, this forum was always helpful to me, it’s great!

You may remove sync=false and try. There is synchronization mechanism in gstreamer and sync=false disables it. Please enable it for a try.

And please execute sudo nvpmodel -m 0 and sudo jetson_clocks, to run system at maximum performance. For checking if there is bottleneck in CPU or other hardware blocks, please execute sudo tegrastats to get system status.


Thank you for the response. I forgot to mention, sync=true makes the video have 1 FPS and no audio. I tried adding PTS and DTS to the buffers, but no luck so far.

You may try gst-launch-1.0 command and check if audio and video are synchronized. OpenCV takes significant CPU usage and it may have impact to performance.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.