Hi,
It looks like videorate plugin does not drop frames properly. There are several properties: videorate and would need other users to share suggestion about how to set the properties.
Another solution is to use appsink plugin. May refer to this sample:
Starvation (?) of gstreamer threads - #11 by DaneLLL