Hi,
This looks advancing and would need other users to share experience.
FYR. There are topics about basic UDP streaming:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
Gstreamer TCPserversink 2-3 seconds latency - #13 by Bazziil
Nvv4l2decoder sometimes fails to negotiate with downstream after several pipeline re-launches - #16 by DaneLLL