You can try to open more log for rtsp module to check if there are some log about dropping frame.
GST_DEBUG=3, rtpjitterbuffer:6,rtspsrc:6 gst-launch-1.0 ...
You can get the pipeline graph for decodebin first by referring to the FAQ. Then build your own pipeline based on this graph.