Hi,
A user has mentioned delay in using tsdemux plugin:
Udp h264 to nv3dsink latency optimization - #7 by davidz0yyl
Probably it is the mechanism in transport stream. Are you able to try other muxer such as matroskamux?
Or try UDP streaming:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL