Hi,
Please try videotestsrc plugin and check the latency. We see neglectable latency in our set up. If you see a larger value, it should be from the network.
If network is good and stable, it should be similar to this:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
Also the camera source can have certain latency in capturing frames.