encoding latency issue

rheeso · January 14, 2020, 6:39pm

We use a camera that is not recognized by ‘v4l2-ctl’, and then link the camera’s own api with the test-appsrc.c code from rtsp-server.
The resolutions used are 1k 1024768 and 2k 20641544. Two tx2 boards are used as encoders and decoders, respectively, and omxvp8enc is used.

Intermittent large encoding time

Encoding time was measured using “MeasureEncoderLatency” option in a system that encodes 1k video and sends it to decoder board via rtsp.
It usually comes out as 3-4ms as below, but sometimes the time is measured irregularly over 10ms.
I have to solve this. Do you have any idea about the cause of this?

KPI: omx: frameNumber= 68 encoder= 3 ms pts= 566666644

KPI: omx: frameNumber= 69 encoder= 3 ms pts= 566666644

KPI: omx: frameNumber= 70 encoder= 3 ms pts= 583333310

KPI: omx: frameNumber= 71 encoder= 3 ms pts= 583333310

KPI: omx: frameNumber= 72 encoder= 3 ms pts= 599999976

KPI: omx: frameNumber= 73 encoder= 3 ms pts= 599999976

KPI: omx: frameNumber= 74 encoder= 3 ms pts= 616666642

KPI: omx: frameNumber= 75 encoder= 3 ms pts= 616666642

KPI: omx: frameNumber= 76 encoder= 3 ms pts= 633333308

KPI: omx: frameNumber= 77 encoder= 4 ms pts= 633333308

KPI: omx: frameNumber= 78 encoder= 14 ms pts= 649999974

KPI: omx: frameNumber= 79 encoder= 4 ms pts= 649999974

KPI: omx: frameNumber= 80 encoder= 3 ms pts= 666666640

KPI: omx: frameNumber= 81 encoder= 3 ms pts= 666666640

KPI: omx: frameNumber= 82 encoder= 3 ms pts= 683333306

KPI: omx: frameNumber= 83 encoder= 3 ms pts= 683333306

KPI: omx: frameNumber= 84 encoder= 3 ms pts= 699999972

KPI: omx: frameNumber= 85 encoder= 3 ms pts= 699999972

KPI: omx: frameNumber= 86 encoder= 2 ms pts= 716666638

KPI: omx: frameNumber= 87 encoder= 3 ms pts= 716666638

KPI: omx: frameNumber= 88 encoder= 3 ms pts= 733333304

KPI: omx: frameNumber= 89 encoder= 7 ms pts= 733333304

KPI: omx: frameNumber= 90 encoder= 13 ms pts= 749999970

No difference in encoding time between using iframe only and not

The system is same as above using 2k resolution.

Using “iframeinterval = 1” option to the encoder, all i frames are used.
In theory, using only i frames should reduce encoding time. However, it is strange that there is no difference between the two latency logs below. Do you have any other opinions about this?

**************************************************************** KPI: omx: frameNumber= 60 encoder= 13 ms pts= 999999990

KPI: omx: frameNumber= 61 encoder= 13 ms pts= 999999990

KPI: omx: frameNumber= 62 encoder= 13 ms pts= 1033333323

KPI: omx: frameNumber= 63 encoder= 13 ms pts= 1033333323

KPI: omx: frameNumber= 64 encoder= 13 ms pts= 1066666656

KPI: omx: frameNumber= 65 encoder= 13 ms pts= 1066666656

KPI: omx: frameNumber= 66 encoder= 14 ms pts= 1099999989

KPI: omx: frameNumber= 67 encoder= 22 ms pts= 1099999989

KPI: omx: frameNumber= 68 encoder= 13 ms pts= 1133333322

KPI: omx: frameNumber= 69 encoder= 14 ms pts= 1133333322

KPI: omx: frameNumber= 70 encoder= 14 ms pts= 1166666655

************************************************************* KPI: omx: frameNumber= 60 encoder= 13 ms pts= 999999990

KPI: omx: frameNumber= 61 encoder= 14 ms pts= 999999990

KPI: omx: frameNumber= 62 encoder= 14 ms pts= 1033333323

KPI: omx: frameNumber= 63 encoder= 13 ms pts= 1033333323

KPI: omx: frameNumber= 64 encoder= 14 ms pts= 1066666656

KPI: omx: frameNumber= 65 encoder= 14 ms pts= 1066666656

KPI: omx: frameNumber= 66 encoder= 13 ms pts= 1099999989

KPI: omx: frameNumber= 67 encoder= 17 ms pts= 1099999989

KPI: omx: frameNumber= 68 encoder= 14 ms pts= 1133333322

KPI: omx: frameNumber= 69 encoder= 14 ms pts= 1133333322

KPI: omx: frameNumber= 70 encoder= 13 ms pts= 1166666655

DaneLLL · January 15, 2020, 1:43am

Hi,
The omx plugins run in dynamic frequency mode. You can apply the patch to enable max clocks.
https://devtalk.nvidia.com/default/topic/1032771/jetson-tx2/no-encoder-perfomance-improvement-before-after-jetson_clocks-sh/post/5255605/#5255605

Since we are deprecating omxplugins. Suggest you use v4l2 plugins. You can set the property dircetly:

_ maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

rheeso · January 15, 2020, 9:51am

Hi, DaneLLL.
I applied your answer.
I used v4l2 plugin and max-perf.
However, Latency is still measured intermittently over 10ms.

The frequency of large delays is reduced initially. However, it still comes out more than 10ms once every 2-400 frames as shown below.

PI: v4l2: frameNumber= 271 encoder= 2 ms pts= 4499999955

KPI: v4l2: frameNumber= 272 encoder= 3 ms pts= 4533333288

KPI: v4l2: frameNumber= 273 encoder= 3 ms pts= 4533333288

KPI: v4l2: frameNumber= 274 encoder= 3 ms pts= 4566666621

KPI: v4l2: frameNumber= 275 encoder= 2 ms pts= 4566666621

<b>KPI: v4l2: frameNumber= 276 encoder= 10 ms pts= 4599999954</b>

KPI: v4l2: frameNumber= 277 encoder= 3 ms pts= 4599999954

KPI: v4l2: frameNumber= 278 encoder= 3 ms pts= 4633333287

KPI: v4l2: frameNumber= 279 encoder= 3 ms pts= 4633333287

At about 5000 frames, large latency is much more frequent than in the early ones.

KPI: v4l2: frameNumber= 5493 encoder= 3 ms pts= 91533332418
[b]
KPI: v4l2: frameNumber= 5494 encoder= 10 ms pts= 91566665751[/b]

KPI: v4l2: frameNumber= 5495 encoder= 8 ms pts= 91566665751

KPI: v4l2: frameNumber= 5496 encoder= 3 ms pts= 91599999084

KPI: v4l2: frameNumber= 5497 encoder= 3 ms pts= 91599999084

KPI: v4l2: frameNumber= 5498 encoder= 3 ms pts= 91633332417

KPI: v4l2: frameNumber= 5499 encoder= 2 ms pts= 91633332417

KPI: v4l2: frameNumber= 5500 encoder= 3 ms pts= 91666665750

KPI: v4l2: frameNumber= 5501 encoder= 3 ms pts= 91666665750

KPI: v4l2: frameNumber= 5502 encoder= 2 ms pts= 91699999083

KPI: v4l2: frameNumber= 5503 encoder= 3 ms pts= 91699999083

[b]KPI: v4l2: frameNumber= 5504 encoder= 10 ms pts= 91733332416

KPI: v4l2: frameNumber= 5505 encoder= 10 ms pts= 91733332416[/b]

KPI: v4l2: frameNumber= 5506 encoder= 3 ms pts= 91766665749

KPI: v4l2: frameNumber= 5507 encoder= 3 ms pts= 91766665749

We want to know the cause of the occasional high latency,
and When time has elapsed, the more often large latency is measured.

DaneLLL · January 16, 2020, 1:51am

Hi,
With ‘maxperf-enable=1’, encoder has been in max clocks always and provides best performance. Your source is RTSP source and it is possible the latency is from the network. You can execute ‘sudo jetson_clocks’, ‘sudo nvpmodel -m 0’ and compare with local video file source:

$ gst-launch-1.0 uridecodebin uri=file:///home/nvidia/jellyfish-10-mbps-hd-hevc.mkv ! nvvidconv ! nvv4l2h264enc maxperf-enable=1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=false  -v