I have a project for the Jetson Nano where I receive AV from HDMI and I try to detect scene changes from audio and video and the length between the scene changes. For this, I need the audio and video to be perfectly in sync (for audio I detect silence and for video I detect dramatic color change).
For this I have 2 pipelines in gstreamer 1.14.5:
pipeline_video_str = ‘v4l2src device=/dev/video0 do-timestamp=true blocksize=64000 ! queue !
image/jpeg,width=1280,height=720,framerate=30/1 ! videorate ! queue !
appsink name=videosink alsasrc device=hw:CARD=MS2109,DEV=0 blocksize=6400 do-timestamp=true ! queue !
audioconvert ! audioresample !
audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! appsink name=audiosink’
This one receives video and audio and sends it to Python callbacks, where I perform the analysis. Then, I forward audio and video to 2 appsrc, to display and play them:
pipeline2_str = ‘appsrc name=videosrc do-timestamp=true format=3 ! jpegparse ! jpegdec ! queue !
appsrc name=audiosrc format=3 do-timestamp=true ! audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! audioconvert !
queue ! alsasink device=hw:0,3 sync=false’
This works fine for a bit,
but after a while 2 things start to happen, which may be strongly linked:
Frames seem to start dropping and AV becomes desynchronized. I notice these problems because I assume I have 30 FPS and, for example, if I detect a 30 seconds length part, I have received less than 900 frames (something like 877-889 frames). The desync is apparent because the scene change detector becomes inaccurate (so audio silence does not line up with color change). I want to mention that the CPU is not maxed out, I perform computation on GPU as well.
I would appreciate help regarding:
- How can I better debug my setup? I tried printing the “drop” property of “videorate”, but it does not seem consistent with what I am experiencing. I want to make sure that my bug happens because of dropped frames.
- Can I improve the pipelines to not drop frames? I tried adding “max-size-buffers=0 max-size-bytes=0 max-size-time=0” to queues, but I still experience issues.
- What is the best approach to keep the synchronization?
Thank you very much for reading, this forum was always helpful to me, it’s great!