Nvv4l2h264enc latency and preset-level

Hi,
What is the relationship between the preset-level and the H264 encoding latency?
This my test results:

preset 0 preset 1 preset 2 preset 3 preset 4
Total frames 409 646 623 568 720
max latency (ms) 112 135 104 121 75
average latency (ms) 15.50 13.83 13.51 15.39 15.17
min latency (ms) 11 10 10 11 11

The test was carried out at the same test conditions except changed the preset-level.
Would you please let me know:

  1. The H264 average encode latency at 14 ms, is correct?
  2. Changing the preset level, I can’t see the encoding latency changes significantly, is it correctly?
  3. If my test output is not correct, what can I do for it?
    Thanks

CX

Hi,
The test result is dependent to resolution and frame rate. If you would like to achieve 4Kp30 encoding , you only can set to UltraFastPreset. If your use-case is single 1080p30 encoding, you may set to other preset to get better image quality.

So if you run 4K encoding and compare the preset levels, should see different result and fps.

We don’t have test data for latency. For achieving low latency, suggest you set the properties:

  poc-type            : Set Picture Order Count type value
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2 Default: 0
  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

For encoding into I P P P … frames, encoder would need to keep one reference frame so you should see one-frame delay. If your source is in 30fps, there is 33ms latency.

Thank you for your kind reply.
My video streaming frame size is 2592 x 1944 less than 4k. frame rate is 24 fps.
This is my test script for OpenCV VideoCapture

v4l2src device=/dev/video0 \
    ! video/x-raw, format=UYVY, width=2592, height=1944, framerate=24/1 \
    ! nvvidconv ! video/x-raw(memory:NVMM), format=I420 \
    ! tee name=camNVMM ! queue ! nvv4l2h264enc preset-level=1 MeasureEncoderLatency=1 maxperf-enable=true insert-vui=true insert-sps-pps=1\
    ! tee name=h264_stream ! queue ! h264parse ! rtph264pay ! udpsink clients=192.168.18.18:38298 \
    h264_stream. ! queue ! h264parse ! matroskamux ! filesink location=test_h264.mkv \
    camNVMM. ! queue ! nvvidconv ! video/x-raw, format=GRAY8, width=640,height=480 ! appsink

maxperf-enable=true.
I don’t know how to set poc-type. Would you please give me your suggestion based on my script above?
Thanks
CX

Hi,
Please try to set poc-type=2

where to set poc-type=2? nvv4l2h264enc has no property of “poc-type”

I have updated MicroSD card image to jetson-nano-jp46-sd-card-image, where I can set poc-type=2.
This is my test result

Sometimes, the encoding latency is more than 2200 ms, which is not acceptable for real-time video capture.

This is my gstreamer nvv4l2h264enc code

nvv4l2h264enc preset-level=1 MeasureEncoderLatency=1 poc-type=2 maxperf-enable=true insert-vui=true insert-sps-pps=1 bitrate=10000000 control-rate=1 vbv-size=734000

what can I do for it?
Thanks

I tested on poc-type=0 and poc-type=1.
The test results are:
image

This is the summery of the test results:

poc-type 0 poc-type 1 poc-type 2
max 2051 3126 2262
min 9 9 9
mean 14.78280178 15.6691747 15.16752005
SD 35.02557257 53.41377167 39.65427069
num of frames 9907 12758 11717

I can’t see any poc-type is acceptable for real time video streaming. It looks like poc-type 0 is better than others?
why the encoding latency more than 2 seconds for any poc-type, event more than 3 seconds for poc-type=1?
Thanks

Hi,
Please check explanation about poc-type in
H.264/AVC Picture Management - Vcodex

Generally it should have low latency in setting poc-type=2. Probably it is not that obvious in gstreamer frameworks. For further enhancement, may try jetson_multimedia_api.

Is there any jetson_multimedia_api sample code available to test nvv4l2h264enc latency?
Thanks

Hi,
All samples are in

/usr/src/jetson_multimedia_api/samples

Please try

01_video_encode$ sudo nvpmodel -m 0
01_video_encode$ sudo jetson_clocks
01_video_encode$ ./video_encode _INPUT_YUV_FILE_ 1920 1080 H264 output.h264 --stats --max-perf -poc 0/1/2

Thank you for your kind reply.
Can I change INPUT_YUV_FILE to the camera source. My camera is e-con system See3CAM_CU55, which support v4l2. If yes, please let me know how. If not, would you please let me know how can I get the video from the camera and save it as the INPUT_YUV_FILE . The camera video format is UYVY. I need to test encode latency at full frame size 2592 x 1944.
Thanks

Hi,
Just noticed we had discussion about the use-case:
Sample code 12_camera_vl2_cuda does not support e-CAM50 at full size

Since the resolution is unique and hits constraint of data alignment, are you able to use standard resolution such as 1920x1080?

No. The project requires to use full size.

Hi,
If the source is not able to meet constraint of data alignemnt, you would need to call Raw2NvBuffer() to copy data from CPU buffer to NvBuffer. This memory copy may trigger certain latency. Due to this there may not be benefit in switching from gstreamer to jetson_multimedia_api, but if you would like to give it a try, may refer to this patch:
tegra multimedia samples not working properly - #11 by DaneLLL

It demonstrates 12_camera_v4l2_cuda + NvVideoEncoder. You would also need to port the options –stats --max-perf -poc from 01_video_encode.