GStreamer stops unexpectedly after getting H.264 for about 7 minutes

When using nvv4l2h264enc with bit-packetization=true, after taking H.264 data for about 7 minutes failed to allocate will occur.

Is there something wrong with my settings?

NG Command and result

$ time sudo gst-launch-1.0 -q videotestsrc is-live=true ! videorate ! video/x-raw,width=1920,height=1080,framerate=15/1 ! nvvidconv ! nvv4l2h264enc iframeinterval=15 bitrate=3000000 bit-packetization=true slice-header-spacing=1024 insert-sps-pps=true insert-vui=true ! video/x-h264,stream-format=byte-stream ! queue ! h264parse ! queue ! filesink location=dump.h264
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA_ENC: bBlitMode is set to TRUE 

(gst-launch-1.0:24534): GLib-ERROR **: 22:53:12.957: ../../../../glib/gmem.c:105: failed to allocate 3110551 bytes
Trace/breakpoint trap

real	7m36.660s
user	2m32.128s
sys	1m43.924s

If I remove bit-packetization=true and slice-header-spacing=1024, the error is not occur.

OK Command

$ gst-launch-1.0 -q videotestsrc is-live=true ! videorate ! video/x-raw,width=1920,height=1080,framerate=15/1 ! nvvidconv ! nvv4l2h264enc iframeinterval=15 bitrate=3000000 insert-sps-pps=true insert-vui=true ! video/x-h264,stream-format=byte-stream ! queue ! h264parse ! queue ! filesink location=dump.h264

Environment

  • Jetson TX2 Developer Kit
  • NVIDIA L4T 32.7.2
$ cat /etc/nv_tegra_release
# R32 (release), REVISION: 7.2, GCID: 30192233, BOARD: t186ref, EABI: aarch64, DATE: Sun Apr 17 09:53:50 UTC 2022

Hi,
Does it happen if you set bit-packetization=false and slice-header-spacing=1024?

Does it happen if you set bit-packetization=false and slice-header-spacing=1024 ?

Yes.

$ time sudo gst-launch-1.0 -q videotestsrc is-live=true ! videorate ! video/x-raw,width=1920,height=1080,framerate=15/1 ! nvvidconv ! nvv4l2h264enc iframeinterval=15 bitrate=3000000 bit-packetization=false slice-header-spacing=1024 insert-sps-pps=true insert-vui=true ! video/x-h264,stream-format=byte-stream ! queue ! h264parse ! queue ! filesink location=dump.h264
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA_ENC: bBlitMode is set to TRUE 

(gst-launch-1.0:9381): GLib-ERROR **: 17:43:19.025: ../../../../glib/gmem.c:105: failed to allocate 3110551 bytes
Trace/breakpoint trap

real	26m30.793s
user	7m57.032s
sys	4m50.720s

It took longer this time.

I can confirm that this happens on Orin running JP5.0.2 (R35.1). You may enable tegrastats monitoring in one terminal, then in a second terminal launch:

# Without nvv4l2h264enc, no RAM usage increase:
gst-launch-1.0 -v videotestsrc is-live=true ! video/x-raw,width=1920,height=1080,framerate=15/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! fakesink

# With nvv4l2h264enc using these parameters, constant RAM usage increase such as memory leak:
gst-launch-1.0 -v videotestsrc is-live=true ! video/x-raw,width=1920,height=1080,framerate=15/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvv4l2h264enc iframeinterval=15 bitrate=3000000 bit-packetization=true slice-header-spacing=1024 insert-sps-pps=true insert-vui=true ! video/x-h264,stream-format=byte-stream ! fakesink

Also, noticed after some time with latter pipeline these unexpected kernel error messages:

[ 1956.186554] nvgpu: 17000000.ga10b nvgpu_cic_mon_report_err_safety_services:92   [ERR]  Error reporting is not supported in this platform
[ 1956.186984] nvgpu: 17000000.ga10b           acr_report_error_to_sdl:53   [ERR]  ACR register access failure
[ 1956.187306] nvgpu: 17000000.ga10b     nvgpu_acr_wait_for_completion:143  [ERR]  flcn-1: HS ucode boot failed, err 1b
[ 1956.187644] nvgpu: 17000000.ga10b     nvgpu_acr_wait_for_completion:145  [ERR]  flcn-1: Mailbox-1 : 0xabcd1234
[ 1956.187982] nvgpu: 17000000.ga10b nvgpu_cic_mon_report_err_safety_services:55   [ERR]  Error reporting is not supported in this platform
[ 1956.188368] nvgpu: 17000000.ga10b nvgpu_pmu_report_bar0_pri_err_status:41   [ERR]  PMU falcon bar0 timeout. status(0x0), error_type(0xc)
[ 1956.188736] nvgpu: 17000000.ga10b            ga10b_bootstrap_hs_acr:72   [ERR]  ACR bootstrap failed
[ 1956.189512] nvgpu: 17000000.ga10b        nvgpu_acr_bootstrap_hs_acr:85   [ERR]  ACR bootstrap failed
[ 1956.191119] nvgpu: 17000000.ga10b       nvgpu_acr_construct_execute:108  [ERR]  Bootstrap HS ACR failed
[ 1956.192722] nvgpu: 17000000.ga10b            nvgpu_finalize_poweron:1010 [ERR]  Failed initialization for: g->ops.acr.acr_construct_execute
[ 1956.238038] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[ 1956.238325] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg

EDIT: Seems

slice-header-spacing=8192

is your best choice, the leak may increase with deviation from this value.

Thank you for the helpful information.

I investigated the memory usage. And I think it’s very possible that it’s a memory leak.
It seems that memory growth cannot be avoided as long as the nvv4l2h264enc is used.

I think the only workaround is to use omxh264enc even though it is deprecated.

I really hope that it will be improved in future releases.


Investigation

Measure memory usage while running the GStreamer for 10 minutes with following command:

gst-launch-1.0 -v videotestsrc is-live=true ! video/x-raw,width=1920,height=1080,framerate=15/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvv4l2h264enc iframeinterval=15 bitrate=${bitrate} bit-packetization=true slice-header-spacing=${size} insert-sps-pps=true insert-vui=true ! video/x-h264,stream-format=byte-stream ! fakesink

Patterns:

  • bitrate=3000000
    • slice-header-spacing=1024
    • slice-header-spacing=4096
    • slice-header-spacing=8192
    • slice-header-spacing=16384
  • bitrate=6000000
    • slice-header-spacing=1024
    • slice-header-spacing=4096
    • slice-header-spacing=8192
    • slice-header-spacing=16384

Measure memory usage:

$ tegrastats 
RAM 627/7858MB (lfb 1466x4MB) SWAP 28/3929MB (cached 0MB) CPU [1%@345,off,off,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@38C MCPU@38C PMIC@50C Tboard@37C GPU@36.5C BCPU@38C thermal@37.7C Tdiode@36.5C
RAM 627/7858MB (lfb 1466x4MB) SWAP 28/3929MB (cached 0MB) CPU [1%@345,off,off,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@38C MCPU@38C PMIC@50C Tboard@37C GPU@36.5C BCPU@38C thermal@37.4C Tdiode@36.75C
RAM 700/7858MB (lfb 1466x4MB) SWAP 28/3929MB (cached 0MB) CPU [13%@2035,off,off,7%@2035,6%@2035,14%@2035] EMC_FREQ 0% GR3D_FREQ 0% PLL@39C MCPU@39C PMIC@50C Tboard@37C GPU@36.5C BCPU@39C thermal@37.4C Tdiode@36.75C
    ~~~
      ^-- measure this

Results of bitrate=3000000

nvv4l2_bitrate-3000000

Results of bitrate=6000000

nvv4l2_bitrate-6000000

Results of omxh264enc bitrate=3000000

Command:

gst-launch-1.0 -v videotestsrc is-live=true ! video/x-raw,width=1920,height=1080,framerate=15/1 ! omxh264enc iframeinterval=15 bitrate=3000000 bit-packetization=true slice-header-spacing=${size} insert-sps-pps=true insert-vui=true ! video/x-h264,stream-format=byte-stream ! fakesink

omx_bitrate-3000000