Hi,
You may set latency=0 to rtspsrc for a try. And can set xvimagesink sync=0 or nv3dsink sync=0 to disable synchronization mechanism of gstreamer.
Can also try this setup for comparison:
Jetson AGX Orin FAQ
Q: Is there an example for running UDP streaming?