Latency when decoding the camera on the same device

I was trying to reduce the transmission latency (280ms) when using udpsink with a sender and receiver pipeline tested using two Xaviers. I tried to decode on the same Xavier and still noticed an approximately 140ms latency with encoding and decoding at the same time. Any ways to reduce that? Any help is much appreciated.
Here are the two pipelines I used for sender and receiver:


gst-launch-1.0 pylonsrc ! 'video/x-raw, width=1920, height=1080, format=YUY2, framerate=60/1' ! nvvidconv ! nvv4l2h265enc bitrate=5000000 insert-sps-pps=1 idrinterval=30 insert-vui=1 maxperf-enable=true ! 'video/x-h265, stream-format=(string)byte-stream' ! h265parse ! rtph265pay mtu=1320 ! udpsink host= port=15000 sync=false async=false -e


gst-launch-1.0 -e udpsrc port=15000 ! 'application/x-rtp, encoding-name=H265, payload=96' ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! autovideosink sync=false async=false

The latency mey be from pylonsrc. Please try this setup:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL

Yo can compare deviation between videotestsrc and pylonsrc to check if the latency is from pylonsrc