Low latency video streaming with H.265

I’m attempting to get very low video latency, say below 80ms, between two TX1 boards using the provided camera. One TX1 displays the video and connected via ethernet, and another TX1 connected via WiFi which will be encoding and sending the video stream. We have it setup like this is because we’ll eventually want to use a TX1 on a hexacopter/drone.

So far I’ve only been able to get the latency down to roughly 1/4th a second using the following commands. Note that I had to build gstreamer myself to get the rtph265pay and rtph265depay elements as discussed on GStreamer RTP H.265 elements missing thread.

On the display side I run the following, via SSH cause it was easier to do hence exporting DISPLAY environment variable:

export DISPLAY=:0
cd ~/gst_1.6.0/out/bin
./gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H265,payload=96 ! rtph265depay ! h265parse ! queue ! omxh265dec ! nvoverlaysink sync=false async=false -e

And on the video encoding / video source side (display side’s IP is 192.168.5.2):

cd ~/gst_1.6.0/out/bin
./gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! omxh265enc low-latency=1 control-rate=2 bitrate=4000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! h265parse ! rtph265pay mtu=1400 ! udpsink host=192.168.5.2 port=5000 sync=false async=false

As you may noticed that I am vertically flipping the video image (nvvidconv flip-method=6 part) cause for whatever reason the camer’s video is coming in flipped and I’m undoing that odd video result.

The overall network latency is low in our test setup since the boards are only maybe 10 feet away from the wireless access point and it’s on channel that doesn’t seem to be congested. We’re seeing an average of 3ms over 50 pings between said boards and naturally an occasional 150ms here and there as you normally do see over wireless blip up.

So there you have it. I am curious if anybody has suggestions or ideas how I go about reducing the latency to hopefully down to our maximum video latency target range.

I’ve never associated wireless with low latency. However, I’m wondering if your data uses a lot of small frames?

Basically tcp with several small amounts of data might delay the frame being sent until either there is a timeout or more data arrives, increasing latency. In the case of larger data (consistent with video) I might expect to instead require fragmenting the frame and then reassembling it (the reverse of waiting for more data). Should your data be large enough, jumbo frames might actually reduce latency. Whether or not you can set jumbo frames on wireless I don’t know, I’ve not heard of anyone trying. You could test it 100% wired first.

The other thing I’d wonder about is if your use can live with some dropped data or not. TCP and reliability actually requires two-way communications to validate whether a resend is needed. On wireless that’s an extra penalty. If you could live with some dropped data, you might try UDP.

How do you measure the latency?

A few things you could do just to get better understanding how different features might affect the latency:

  1. Maximise CPU/GPU/EMC clocks.
  2. Try 720p and 480p to see if it affects latency.
  3. Try without nvvidconv flip-method=6 (You could that maybe more easily on the rendering side anyway)
  4. Try rtpjitterbuffer instead of queue.

I’m streaming H264 on Jetson TK1 from an RC car and I’m also very interested about the latency. I got something like 150ms end-to-end (I used a blinking LED on front of the camera and measured with light sensors the time between the LED turning on and when the monitor shows it).

I have found the rtp payloader/depayloader can add significant buffering / latency. At least with H264 I had a much lower latency pipeline using an mpegts container, aggregating mpegts packets to the limit of a UDP frame (see: http://tipok.org.ua/taxonomy/term/125) and sending them directly over UDP without any RTP wrapper.

May be worth a shot?

Has anyone tested latency for encoding 1080p60fps (YUV422) to H.264/H.265 on TX1? in ms (miliseconds)?

Hi,

I tested with:

1080p30
H264
Bitrate 4Mbps
Latency=172ms

1080p30
H265
Bitrate 4Mbps
Latency=258ms

In case of H265 it is important to mention that the same Jetson TX1 was generating and consuming the stream.

About how to measure the latency basically I captured the stopwatch rendered in my laptop and displayed the captured stream in other, then I took a picture and subtracted the values. In this case I would think that the payloader adds latency, I wonder if one could optimize it using the NEON.

I saw in the wiki you mentioned there is no gstreamer H265 decoder for the PC… you can actually get a good decoder from the libav gstreamer integration: avdec_h265. I know it is available as of (at least) the 1.7.2 binaries.

Also, gstreamer 1.8.0 has mpegts support for H265 with proper UDP packetization:

Server:

gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \
! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh265enc low-latency=1 control-rate=2 bitrate=4000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! \
h265parse ! mpegtsmux alignment=7 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

Client:

gst-launch-1.0 udpsrc port=5000 ! video/mpegts ! tsparse ! tsdemux ! h265parse ! queue ! avdec_h265 ! xvimagesink sync=false async=false -e

Great! I will give it a try and add it to the wiki. Thanks again!

Hi DavidSoto!

How did it go with these commands? Were you able to get under 100 ms latency?

Hi,

I didn't have the change to test the new pipeline, we have been working in other tasks assigned by our customers :S

-David

I am trying this now, somewhat a different scenario. Using 4 aggregated cellular LTE links, 6 cameras and TX2. Will report back here. Need to find a good way to measure latency.

I am trying:

  1. Over aggregated 4 carrier LTE cellular link
  2. Over local WIFI
  3. Streaming from one Jetson to 6 monitors on one PC vs. to 6 independent Raspberry Pi3’s

Just using one 1080p stream, visually, comparing to our atomic clock it ‘looks’ to be 500ms.

SERVER IS:
gst-launch-1.0 v4l2src device=/dev/video2 ! nvvidconv flip-method=2 ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1’ ! omxh265enc control-rate=2 bitrate=4000000 ! ‘video/x-h265, stream-format=(string)byte-stream’ ! h265parse ! mpegtsmux alignment=7 ! udpsink host=192.168.60.6 port=5000 sync=false async=false

CLIENT IS:
gst-launch-1.0 udpsrc port=5000 ! video/mpegts ! tsparse ! tsdemux ! h265parse ! queue ! avdec_h265 ! autovideosink sync=false async=false -e

Hi Peter

You can measure glass-to-glass latency like this: https://devtalk.nvidia.com/default/topic/1026587/jetson-tx2/csi-latency-is-over-80-milliseconds-/

Hi njal, Thank-you!

Hi peter6v4yn,

Which LTE modems/dongle are you using with Jetson? Do you know of a Jetson-compatible LTE USB Dongle that works in Europe? Nvidia doesn’t seem to “officially” support any cellular modem/dongle.

Hi

We are using Mikrotik devices for routing - they support a wide range of LTE modems.

Hi njal,

Which model specifically? I found only one LTE product in their catalog: https://mikrotik.com/product/sxt_lte_kit#fndtn-specifications

Is it a USB Interface to the Jetson or Ethernet? Did you try to aggregate two or more cellular connections with your setup?

https://wiki.mikrotik.com/wiki/Manual:Peripherals

Jetson is connected via Ethernet.

What does your setup look like? The link you posted points to an OEM board that supports different cellular/wifi modems/dongles.

So is your setup something like: Jetson <-eth-> RouterBOARD <-eth/usb-> 2+ LTE Dongles/modems ?

Did you try directly connecting the Jetson to two LTE USB Dongles (via a hub) and making link aggregation on the Jetson?

We use

Jetson <-eth-> RouterBOARD <-usb-> 1 LTE Dongle

If you want to aggregate LTE - then I recommend

Jetson <-eth-> Pepwave MAX

Thanks njal!

The Pepwave modems are prohibitively expensive (2.4k EUR for the Mini version). So I have to figure out a way to do it in software.

Did you try directly connecting the LTE dongle directly to the Jetson?