Using nvv4l2h264enc encoder memory increases when releasing pipeline and setting up a new one

gikniz · October 7, 2019, 1:26pm

Using GStreamer 1.16.1 to encode video stream from a 4K camera feed, we noticed that each time we tear down the pipeline and setup a new one, the system memory use increases.

We nailed down the problem to show up specifically when using nvv4l2h264enc encoder (instead of x264enc).

The following code reproduces the issue on Xavier… For instance, running about 100 iterations of the test increases memory usage of the system of 20% in my experiments, but the memory used is not reported to be consumed by the test application in the System Manager (and I could not find which process it is accounted to, but the memory use keeps increasing without bounds).

Thanks in advance for your feedback!

(Sample code to repro is coming in attachment)
POC.zip (6.39 KB)

DaneLLL · October 8, 2019, 2:26am

Hi,
The gstreamer version in r32.2 is 1.14.5. Does it happen with the default version?

gikniz · October 8, 2019, 3:04pm

Hi,

Thanks for this advice. It was worth to try but using GStreamer 1.14.5 (flashing the device), I get the same memory build up.

I also notice the same error messages show up starting during iteration 122 (100% repro) and stalling in executing iteration 123 (I don’t know on which gstreamer call yet)

Command line:
GST_TRACERS=“leaks” GST_DEBUG=“GST_TRACER:7” ./kogniz_tests --gtest_filter=Saving --gtest_repeat=-1

[----------] Global test environment set-up.
[----------] 1 test from Gstreamer
[ RUN      ] Gstreamer.SavingVideo
| Memory t0|     99256
Opening in BLOCKING MODE  
NvMMLiteOpen : Block : BlockType = 4  
===== NVMEDIA: NVENC =====
Frame #  0/300
NvMMLiteBlockCreate : Block : BlockType = 4  
H264: Profile = 66, Level = 0  
Frame # 25/300
Frame # 50/300
Frame # 75/300
Frame #100/300
Frame #125/300
Frame #150/300
Frame #175/300
Frame #200/300
Frame #225/300
Frame #250/300
Frame #275/300
Frame #300/300
GST_MESSAGE_EOS!
| Memory t1|     99256/    196328, DELTA: -         0
[       OK ] Gstreamer.SavingVideo (5592 ms)
[----------] 1 test from Gstreamer (5592 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (5592 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 122) . . .

Note: Google Test filter = *Saving*
[==========] Running 1 test from 1 test case.                                                                                                                                                                                                                                   
[----------] Global test environment set-up.
[----------] 1 test from Gstreamer
[ RUN      ] Gstreamer.SavingVideo
| Memory t0|     99256
Opening in BLOCKING MODE  
NvMMLiteOpen : Block : BlockType = 4  
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4  
Frame #  0/300
H264: Profile = 66, Level = 0  
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
Frame # 25/300
[...]
Frame #300/300
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 30, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
GST_MESSAGE_EOS!
| Memory t1|     66912/    196328, DELTA: -     32344
[       OK ] Gstreamer.SavingVideo (5309 ms)
[----------] 1 test from Gstreamer (5309 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (5310 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 123) . . .

Note: Google Test filter = *Saving*
[==========] Running 1 test from 1 test case.                                                                                                                                                                                                                                   
[----------] Global test environment set-up.
[----------] 1 test from Gstreamer
[ RUN      ] Gstreamer.SavingVideo
| Memory t0|     66912
Opening in BLOCKING MODE  
NvMMLiteOpen : Block : BlockType = 4  
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4  
Frame #  0/300
H264: Profile = 66, Level = 0  
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 28, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)

DaneLLL · October 9, 2019, 8:17am

Hi,
You may also try omxh264enc.

gikniz · October 9, 2019, 2:15pm

I forgot to mention, but I tried omxh264enc also prior to post this request.

It behaves the same from the memory perspective and starts the misbehavior during the same iteration (122) of the test (the exact same way). The only difference is that it does not hang, so the NvRmChannelSubmit error keeps repeating and memory consomption keeps building up until the oom-killer kicks in.

DaneLLL · October 14, 2019, 6:19am

Hi,
We have tried simple pipeline launch and do not observe the issue:
[url]https://devtalk.nvidia.com/default/topic/1026106/jetson-tx1/usage-of-nvbuffer-apis/post/5219225/#5219225[/url]

Please check if you miss to unref certain objects in your sample.

gikniz · October 28, 2019, 1:41pm

I tested using the simple pipeline launch sample code you suggested (modifying the pipeline description…) and reproduced the problem in there as well (it takes about 246 iterations).

Further investigations revealed that the leak occurs when using the nvvidconv element.
The leak involves memory and also shows up as one dmabuf not released each time the pipeline is setup and tear down.

I also understood that there is no way to avoid using this converter when using the appsrc to feed the nvv4l2h264enc or feeding the appsink from the nvv4l2decoder.

Using omxh264enc/omxh264dec does not require to use the nvvidconv, but therefore using videoconvert instead ends up obviously to load the CPU.

I confirm there is no unref missing in my sample, by code inspection for one thing, but also using GST_TRACERS=“leaks” GST_DEBUG=“GST_TRACER:7” when executing the sample code.

DaneLLL · October 29, 2019, 2:25am

Hi,
Please share the modified cpp so that we can build/run to reproduce the issue.

gikniz · November 5, 2019, 7:38pm

Here it is.

Running this sample code, the memory usage of my device goes from 3.9 to 10 Gb, and hangs during LOOP iteration #124.
TestNvidiaSampleMain.cpp (4.6 KB)

DaneLLL · November 6, 2019, 2:32am

Hi,
Thank you. We will try to reproduce the issue.

DaneLLL · December 11, 2019, 6:28am

Hi,
We can reproduce the issue and are checking further. Will update once there is new finding. Thanks.

DaneLLL · December 18, 2019, 8:46am

Hi,
We have fixed a leak in running:

video/x-raw,format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420'

Please try the attachment on r32.2.x.
r32_2_x_TEST_libgstnvvidconv.zip (22.8 KB)

nr94 · December 26, 2019, 2:24pm

Hello, is this fix rolled into Jetpack 4.3?

DaneLLL · December 26, 2019, 11:41pm

Hi,
It does not catch Jetpack4.3(r32.3.1). We will provide a prebuilt lib for r32.3.1.

DaneLLL · December 30, 2019, 3:10am

Hi,
For Jetpack4.3(r32.3.1), please use the attachment.
r32_3_libgstnvvidconv.zip (23.1 KB)

nr94 · December 30, 2019, 4:29pm

Will this be made available in the new Jetson package repo?

DaneLLL · December 31, 2019, 1:29am

Hi,
It will be in next r32.4. Since r32.3.1 is just released, there may be an interval of few months.

drewqm96r · January 2, 2020, 9:53pm

video/x-raw,format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420'

Is this the only use case that was fixed and tested? We don’t use RGBA in our pipeline but we do use the nvvidconv and nvv4l2h264enc elements. Is it worth replacing our .so with the new .so if we don’t use RGBA?

DaneLLL · January 3, 2020, 1:44am

Hi,
The fix is for all video/x-raw → video/x-raw(memory:NVMM) conversion.

listsw5yds · April 16, 2020, 10:09am

Hi,

I’m experiencing similar issues for conversion from “video/x-raw(memory:NVMM),format=I420” to “video/x-raw,format=RGBA” after decoding using nvv4l2decoder. I have errors like these ones when using “nvvideoconvert”:

NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)

and errors like these ones when using “nvvidconv” (with the patch available here applied):

NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 12,

is there a patch for this too? thanks