Hi,
It seems like you have customization to the power mode. By default we have 6 cores@1.4GHz and 2 cores@1.9GHz. Your tegrastats shows 6 cores@1.9GHz. Maybe you can try default power modes.
And may try
taskset(1) - Linux manual page
to schedule gst-launch-1.0 commands to individual cores.
Adn can compare with videotestsrc to check if the latency is from the source:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL