Hi,
As comparison, please also try gst-launch-1.0 commands in
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
See if you can achieve the same performance. And then can use videotestsrc to simulate your source like:
$ gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,format=GRAY16_LE,width=finalWidth,height=finalHeight ! ...