I think i also had encounter the same problems relating to the frame rate too.
My pipeline like this: rtspsrc ! decodebin ! nvvidconv ! nvvideosink, and i get the frames via EGL stream with mailbox mode.
As time goes on, video delay grows, and the process’s memory also grows.
The camera output 2048x1536 h264 video stream at about 25fps, and the camera and tx2 both connected to the same ethernet, the stream bitrates about 3Mbps, so the network latency can be excluded.
I examine the usage via tegrastats, the CPU/GPU/NVDEC reports low usage. After running for some time, the EGL stream state often not in EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR state.
So, i think the problem is the decoder output lower frame rate than actually is.