Hi,
This may not be the cause of latency. It is very likely from the source or encoder setting. Please try the steps in this post and check if you can get identical result:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
And set insert-sps-pps=1 idrinterval=15 in your use-case and try.