Gstreamer sync between video and audio

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi,

I have this pipeline in gstreamer that takes video and audio from an hdmi capture card:

pipeline_video_str = 'v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! queue ! \
                    videoconvert ! video/x-raw,format=RGB ! appsink name=videosink alsasrc device=hw:CARD=MS2109,DEV=0 blocksize=6400 ! queue ! audioconvert ! audioresample ! \
                    audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! appsink name=audiosink'

and these 2 callbacks for audio and video. You can see I put in 2 shared queues the chunks and the PTS.

def on_new_sample(sink):
    global qVideoShared

    sample = sink.emit('pull_sample')
    buf = sample.get_buffer()
    result, mapinfo = buf.map(Gst.MapFlags.READ)
    data = mapinfo.data
    size = mapinfo.size
    frame = np.ndarray((720, 1280, 3), buffer=data, dtype=np.uint8)

    qVideoShared.put((frame, buf.pts / Gst.SECOND))

    buf.unmap(mapinfo)
    return Gst.FlowReturn.OK

def on_new_audio_sample(sink):
    global qAudioShared

    sample = sink.emit('pull_sample')
    buf = sample.get_buffer()
    result, mapinfo = buf.map(Gst.MapFlags.READ)
    data = mapinfo.data
    size = mapinfo.size
    frame = struct.unpack('%df' % (size//4), data)
    qAudioShared.put((frame, buf.pts / Gst.SECOND))

    buf.unmap(mapinfo)
    return Gst.FlowReturn.OK

I want to make sure I correctly sync a frame with its audio chunk when I get the elements from qAudioShared and qVideoShared. How can I achieve this? PTS seems to be completely useless, as it’s the same for both audio and video, even at desynchronization. If I perform a CPU stress test during runtime, the video becomes desynchronized (if I write 30 seconds of content on disk, the video actually speeds up, probably because of dropped frames). I don’t understand why PTS does not help me here. Any help is appreciated.

Thank you!

3

Hi,
Please set the property to v4l2src and alsasrc and check if PTS values are expected:

  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: false

And instead of using appsink, you may try to save to a file and check if audio/video is synchronized.

4

Thank you for the response. I try adding do-timestamp, but I have the same result, the PTS are the same for video and audio, even though the video is dropping frames and becomes desynchronized.

I performed 2 tests with saving to file:

gst-launch-1.0 -e v4l2src device=“/dev/video0” ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw,format=RGB ! mux. alsasrc device=hw:CARD=MS2109,DEV=0 ! queue max-size-buffers=1000 ! audioconvert ! audioresample ! voaacenc ! aacparse ! queue ! qtmux name=mux ! filesink location=test.mp4

gst-launch-1.0 -e v4l2src device=“/dev/video0” ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! nvvidconv ! “video/x-raw(memory:NVMM),format=NV12” ! nvv4l2h264enc! h264parse ! mux. alsasrc device=hw:CARD=MS2109,DEV=0 ! queue max-size-buffers=1000 ! audioconvert ! audioresample ! voaacenc ! aacparse ! queue ! qtmux name=mux ! filesink location=test.mp4

The first one uses raw RGB and it seems to have the same issues with my pipeline. The h264 one runs perfectly though. Maybe I need to write h264 to appsink and convert to RGB in Python? I’m not sure how to proceed further.

5

I changed to receiving jpeg in Python and seems to be better:

v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 !
appsink name=videosink alsasrc device=hw:CARD=MS2109,DEV=0 blocksize=6400 ! queue ! audioconvert ! audioresample !
audio/x-raw,format=F32LE,layout=interleaved,rate=44100,channels=1 ! appsink name=audiosink’

So raw RGB was causing massive frame drops and desync I guess.

6

Hi,

This is possible since RGB data is processed on CPU. The CPU capability may dominate performance. Please execute sudo jetson_clocks to run CPU cores at maximum clock. To see if it can achieve target performance.

7

Thank you so much for your help, your idea with writing to file helped me arrive at the RGB conclusion. Now I use turbojpeg to decode to RGB in Python and the desync seems to be gone on gstreamer side. We can close this.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.