Video streaming improvement questions

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hello,

We are looking at setting up low latency streaming with the help of Nvidia Nano devices.
In order to benchmark performance, we have set up gstreamer pipelines to measure glass to glass latency.
The test environment consists of 60Hz display connected through HDMI and a raspberry pi v2.1 sensor connected to the Nano.
By running the desired pipelines however we got latency times as follows:

Video codec latency measurement:
camera at 120fps: 61ms
gst-launch-1.0 nvarguscamerasrc maxperf=true timeout=30 ! ‘video/x-raw(memory:NVMM), width=1280, height=720, framerate=120/1, format=NV12’ ! nvv4l2h265enc maxperf-enable=1 preset-level=1 bitrate=8000000 iframeinterval=120 ! nvv4l2decoder ! nvoverlaysink -e -v

camera at 60fps: 92ms
gst-launch-1.0 nvarguscamerasrc maxperf=true timeout=30 ! ‘video/x-raw(memory:NVMM), width=1280, height=720, framerate=60/1, format=NV12’ ! nvv4l2h265enc maxperf-enable=1 preset-level=1 bitrate=4000000 iframeinterval=120 ! nvv4l2decoder ! nvoverlaysink -e -v

Video preview latency measurement:
camera at 120fps: 155ms
gst-launch-1.0 nvarguscamerasrc maxperf=true timeout=30 ! ‘video/x-raw(memory:NVMM), width=1280, height=720, framerate=120/1, format=NV12’ ! nvoverlaysink -e -v

camera at 60fps: 61ms
gst-launch-1.0 nvarguscamerasrc maxperf=true timeout=30 ! ‘video/x-raw(memory:NVMM), width=1280, height=720, framerate=60/1, format=NV12’ ! nvoverlaysink -e -v

Please help us with the following questions:

  1. The latency measured for video codec is very good, however according to our calculation it would be possible to achieve even better latency. Our formula would be: 8.33(1 frame at capture)+8.33(at encoder)+8.33(at decoder)+16.67(display) = 41.67ms. Would it be possible to increase the gst pipeline performance even further ? Which element introduces the latency unaccounted for ?
  2. The latency measured for preview pipeline is surprisingly much worse at 120fps than the video codec performance. Why would that be ?
  3. Could an application based on jetson_multimedia_api provide lower latency compared to gstreamer plugins ?
2

Hi,

In camera capture, the delay is ~3-4 frames.
In video encoding and decoding, it depends on encoding/decoding group of pictures. For encoding into(or decoding from) I P P P…, the reference frame number is 1 and the latency is ~2 frames.

Could you try with sudo jetson_clcoks and set sync=false in nvoverlaysink. There is synchronization mechanism in gstreamer frameworks. Let’s disable it and check if there is improvement.

Yes, it should give better fprormance to use tegra_multimedia_api.

3

Below case could be limited by the display cap. Have a check with fakesink or xvimagesink

  1. The latency measured for preview pipeline is surprisingly much worse at 120fps than the video codec performance. Why would that be ?