Hi,
The latency may be from camera, encoding, network. Please try this set up:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
To see what latency is with videotestsrc, and then try nvarguscamerasrc. By comparing the result of videotestsrc and nvarguscamerasrc, you can get latency of camera.
Also you can check glass-to-glass latency. Can refer to
Jetson Xavier AGX glass to glass latency