How to use mmapi to achieve video synchronization

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX1
1

I am decoding and display using mmapi, how to achieve video synchronization?
For example, I set a reference clock, and then every frame of data with a timestamp, the driver automatically synchronize video display speed for me.
Just like nveglglessink’s sync function.

Looking forward to your reply

2

Hi Li,
Setting fps to renderer should work:

ctx->renderer->setFPS(ctx->fps);

Do you observe issues with setting the parameter?

3

Hi DaneLLL,
Thank you for your reply.
Set the FPS will make the playback more smooth, but can not use the video encoding timestamp to achieve more accurate audio and video synchronization. Will this feature be added in mmapi in the future?

4

We will evaluate the feature of timestamp-based rendering.

5

Hi Li,
Please share how you implement audio playback along with mmapi. Do you use alsa?

6

Hi DaneLLL,

We have not yet begun to achieve real-time audio decoding part, according to the design, should use gstreamer + ALSA interface.

In order to achieve video and audio synchronization, I was designed:

1, we need a reference timer, this reference time can be the audio timestamp, because the audio packet network transmission is relatively smooth, when the audio using ALSA playback, we will save the timestamp as the reference time.

2, when the audio is played again, we compare the time of the reference timer with the timestamp of the audio package, if the difference is not very large, we do not need to adjust the timer, if the difference is large, we re-adjust the timer, Make sure that the reference time timer is exactly the same as the audio timestamp

3, because the reference timer is based on the audio timestamp, so after the arrival of the audio package, without any delay and buffering, direct decoding to play

4, when the video packet arrives, we use this reference timer to synchronize the video, each time you want to call render(buffer->planes[0].fd) .We use this video frame’s timestamp and reference time for comparison,
If we find this is a future frame, we wait for a while,
If this is a past frame, directly rendering,
If that a long time has passed, discard it.

If mmapi designed the video and audio synchronization interface, I hope that through this interface, set the reference timer directly, and then I just pass the video timestamp to the videodecoder or renderer, mmapi can automatically help me to sync the video frame to the reference time

I think there should be a better way to synchronize, I just share my own way