Udp h264 to nv3dsink latency optimization

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

There seem to be a lot of latency (many frames) when using the nv3dsink to display h.264 coming over UDP. It is an realtime application. Smoothness is not an issue, but latency is a big problem. I want the incoming packets to render at once. As soon as a frame is complete it should be displayed. My current gst pipe:

gst-launch-1.0 udpsrc port 15002 ! tsdemux ! tee name=t ! h264parse ! omxh264dec ! nv3dsink window-x=0 window-y=0 window-width=1920 window-height=1080 sync=false t. ! queue ! multifilesink

  1. How can it be improved in term of latency (optimally remove all buffering but the current incomplete frame)?
  2. Where are the frames getting buffered in this pipe?
  3. Is there a way to inspect the pipe while running to discover where buffering is taking place?
#3

Hi,
The issue may not be in decoding/rendering. Decoder has to receive SPS/PPS and first complete IDR frame and then start decoding. It may be the IDR interval is too big in the h264 stream. For example, if IDR interval=60 frames, you will see maximum 2 second latency in 30 fps case. Suggest you check if you can set the source to IDR interval=15 for a try.

Delay in video
#4

Settind IDR to 1 still gives the same latency. N.B. I do not care for startup latency, even if it is 2 seconds. I do want the streaming latency, once it is up and running, to be as small as possible. I want to disable any form of waiting or caring about fps and just render incoming frames as soon as possible. I use sync=false on the nv3dsink which seem to achieve this to some extent, but still there seem to be some frames buffered (I am not certain in which of the steps in the pipe this happens). Is it possible to set up the pipe to buffering only the frame currently incomplete?

#5

You may try to add option disable-dpb=true to your decoder.

That being said, it is possible that you run into an issue with the kernel socket max buffer size. The values below may be oversized for your case, but you may try these and see if it improves.

sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.rmem_default=33554432
1 Like
#6

Hi,
Please try Honyer_patouceul’s suggestion. Setting disable-dpb=true shoudl help, but not all kinds of h264 stream works with the setting. For some sort of stream, decoder has to keep certain amount of reference frames.
You may try sudo nvpmodel -m 0, sudo jetson_clocks to run in max performance, and nvv4l2decoder enable-max-performance=1 .

#7

I suspect that the latency problem originates in the tsdemux. It seems to have a default latency setting of 700 ms, and can not be changed as far as I understand. Is there any alternative to using the tsdemux?

It seems later versions of the tsdemux plugin support latency setting. How do I go about upgrading the gstreamer plugins on a target?

After doing:

sudo apt install openssl-dev
sudo gst-install --prefix=/home/ubuntu/gst --version=1.16.2

I was able to install gstreamer 1.16.2 according to the user manual. But this version did not have the latency property for tsdemux either. I also failed to install gstreamer 1.18.1 in the same way. It failed with:

sudo gst-install --prefix=/home/ubuntu/gst --version=1.18.1
…
/usr/bin/gst-install: line 82: ./configure: no such file or directory
make: *** No targets specified and no makefile found. Stop.

1.18.0 and 1.18.1 fails in the same way. 1.16.x works.

Delay in video
#9

Hi,
The NVIDIA plugins are built with 1.14.5 and if you upgrade to version far from it, it may fail due to significant deviation in header files.

#11

looked at Index of /src/gstreamer source code for 1.18, it doesn’t have configure file as 1.16 does. So it won’t work if you use gst-install to upgrade (cat /usr/bin/gst-install you can see what exactly it’s doing). You might have to look at the readme file in 1.18 and manually compile it. GitHub - GStreamer/gst-build