omxh264dec/nvvidconv: 34.7MB per minute memory leak/crash when decoding Interlaced H264 video

nvv4l2decoder with enable-max-performance enabled results:

Problem remains

The decoding degraded to 1X (lower than realtime) after decoding 155,230 1080i50 fields (00:52:21.34)

Memory: At boot 367MB, Decoding start: 463MB, ended at 615MB after the 155,230 fields decoded. (152MB leak)

The Jetson Nano was connected to a 5V 4A power supply using the Barrel, jetson_clcoks enabled full power.

Thermal 26C while decoding, 23.5C before the test.

Remarks: One of the CPUs remained to 100% while the others ran at 1%~3%

Adding up to the record:

I ran the same test using the same video content (But encoded as progressive this time) and the system doesn’t show this problem.

Memory footprint 363MB before test. 437MB while running and grow to 459MB after 269,955 Progressive Frames decoded (3 hours of video decoded) and relaxed to 435MB after the program ended execution. (8MB leak per hour of 1080p video decoded.

Decoder performance 257Fps and 280Fps with jetson_clocks

Temperature GPU/CPU 27C

CPUs 13%,7%,9%,7%

This issue may be related to the one I wrote about a couple of weeks ago here: [url]https://devtalk.nvidia.com/default/topic/1055089/jetson-nano/extremely-unreliable-hw-encoder/post/5350167/[/url]

I couldn’t identify the source of the problem, however I could confirm that it exists on progressive inputs as well.

Not to bear any hard feelings, but I find it ironic that on my topic I was told to use omx and here OP was told otherwise…

On a side note, there was recently an update to gstreamer 1.14.4 (previous was 1.14.1) however nvv4l2 codecs all report version 1.14.0 so I doubt something changed there…

Hi 11wallace11,
Your issue is investigated and shall be fixed in next r32.2. If you are going to production based on r32.1 and cannot wait for r32.2, please let us know more about your project.

For v4l2 plugins, we download
https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.14.0.tar.xz
and modify to support hardware acceleration. The modification is open source at
https://developer.nvidia.com/embedded/dlc/l4t-sources-32-1-jetson-nano
gst-v4l2 of r32.2 stays on 1.14.0. If you have seen critical issues being fixed in new 1.14.2(or newer). Please kindly note us.

The issue I’m reporting is on the decoder side. The encoder (As a element) is working fine for me excepting for a bug on the b frames on H264 that I’m going to post in a few minutes. My testings are 3 days of content long and is stable (no memory leaks and reliable)

What you might have experienced is that Nvidia decided to use a low level media engine such as GSTreamer. GStreamr have a prototype tool called gst-launch, this is tool doesn’t handle many of the stuff that makes a commercial product reliable, such drop of connections and recovery, glitches on the stream, lack of standard complains by some vendors, buffer handling, etc. instead you need to build your own app using this high-level media abstraction. or like me, drop all this open source stuff and build your own SW using C++ and the NMAP components. (NMAP as a NVIDIA Multimedia Applications)

*full disclosure, I don’t work for Nvidia and I hope it helps…

Thank you both for taking your time and commenting about my issue in this thread.

@DaneLLL

Thanks for your kind offer and pointing me in the right direction to look for the source code.
I’m not in a hurry and can definitely wait for the next version to come out :)
However, if there’s a public beta or beta-testing program I’ll be more than happy to participate and issue detailed bug reports.

@Rudoplh

I’ve been involved in the “media codecs” scene for more than 10 years now, and while I probably have the basic knowledge necessary to build something that’d work, I feel that it’s not worth my time to start working on a new project for a platform that’s being dominated by Nvidia…
That being said, if Nvidia had a proper git repository somewhere and accepted participation of developers not from the company, it may be a good incentive for me (and probably other developers as well) to participate.
To sum up my stance since it’s probably still not so clear at point, I prefer an open repository that everyone worked on (or forked) instead of the current “open source” form of “here, take the source code and develop something for yourself, and yourself only”.

I’m terribly sorry for deviating so much from the original topic. Feel free to ignore my comment :)

Hi,
On r32.2, comment #5 is a known issue. FYI.

Hi Dane,

The problem on the decoding remains, In fact it went much worst than the status of r32.1.

r32.2 and r32.2.1 which are the same gst-nvvideo4linux2_src code, made the decoding of H264 on the Jetson Nano totally useless. It has being almost 4 months and the problem rather than get fixed went much worst and we have commitments to fulfill.

omxh264dec there is a memory leak, but at least we know that the Jetson is able to play H.264 interlace properly. The memory leak appears to be because the videodecoder Sub-class should implement drain()

The new nvv4l2decoder is a disaster. Rather than fix bugs, It looks like nVidia decided to add more features like a new buffer method and the support of DIVx, something that probably nobody uses since more than 10 years ago (Probably pirates)

The new problem is that the decoding of H264i it is always running like in sync=false, in other words it is playing super fast and without timing control, or the timing of the output is in fields?. In addition there is some variations of H264, that doesn’t play at all (I can share professional stream examples), in other samples it looks like the chroma planes are gone. This problems are not happening on omxh264dec. The caps reported by the decoder are not right they used to report progressive output but now is “mixed.” This can be fixed by putting back a pice of code that was moved. See the picture.

Moving forward: As developers, we are relying on nVidia as a product, as developers we cannot move out of the BETA stage and we need Nvidia to understand that this is not right and is costing us business.

I like @11wallece11 ide to put the gst-nvvideo4linux2_src repo online, maybe at nvidia · GitLab or NVIDIA Corporation · GitHub, so we can help with testing and development in a more agile way.

Adding some note to my last replay:

Part of the memory leak, the Cache part can be fixed by doing

sysctl vm.drop_caches=3

nvv4l2decoder: Have other broken parts, Per example the disableDB=true is producing problems too. The new bufferAPI feature, seems to have no effect into the pipe.

I will he happy to help by making a report and a full test of your software. BUT to be honest, I’m disturbed by how nVidia is releasing software without proper testing or quality control, or adding a note of the know issues and potential fixing schedule.

p.s. I can also put the repo into GitHub, as long nVidia is committed to put the daily changes on that repo to allow other developers help you.

Hi Rudoplh,
We have debugged further and tried similar set up with odroid board (which uses gst-v4l2 open source plug-ins) and find similar observations of increasing vmRSS during playback of long duration interlaced streams. And we are checking to file one bug to the gstreamer-plugins-good community as this seems to be a open source related issue.

For the issue about setting ‘disableDB=true’, if it is a separate issue from decoding interlaced streams, please start a new post for it. Thanks.

Hi Dane,

It cannot be a gstreamer issue.

  1. The memory leak of the omxh264dec is probably related to a warning that gstreamer is reporting s a FIX ME! because the videodecoder Sub-class should implement drain(). I know that someone at nVidia decided no abandon this to focus on nvv4l2decoder. In this case nVidia I believe it will be very helpful for ALL of us that nVidia release the repo into gitLab or GitHub, so we can track the changes made and continue with this effort, as it looks like is easier to fix that the new nvv4l2decoder

  2. nvv4l2decoder. Is working “more or less” ok on 32.1, it is seriously BROKEN on 32.2 and 32.1. This cannot be gstreamer fault.

PLEASE, I know that your group at nVidia is a very small, BUT this group cannot just dismiss this problems and leave your customers in the limbo. Each release posted to the public requires serious testing before the code is release it. It has being months, and we have made commitments and serious investments believing that nVidia released a PRODUCT.

PLEASE, PLEASE, check this issue, It is not gstreamer, the code r.32.2 and r32.2.1 have issues as I pointed, and this code is made by nVidia, no gStreamer.

BTW: The issue is not on long durations streams, in less than few seconds, you will see the memory leak issues.

Hi Rudoplh,
For interlaced-stream decoding, NVIDIA core teams are clarifying the issue with gstreamer community:

Will update once there is further new finding.
We are also reviewing test cases about nvv4l2decoder.

They are waiting for your support, it looks like you are passing fields rather than frames

However, this doesn’t resolve the rest of the issues. Please read back what I reported and needs fixing on your side.

Hi Rudoplh,
We can observe the issue of H264i decoding and make a new topic for tracking:
https://devtalk.nvidia.com/default/topic/1066497/

For the issue ‘the disableDB=true is producing problems too’, is it also specific to H264i stream? Please share more information about this issue so that we can reproduce it. Thanks.

Hi Dane,

Testing video can be a complicated thing. I’m helping Jiang on the FFmpeg-jetson project. I’m developing a series of scripts that I will be happy to share with you, so the release of this SW is tested before is released.

Onething I’m still waiting respond from you is to release this open-source code into nVidia’s GitHub or GitLab sites, so we can help you with test and bug reporting and to avoid the costly waiting.

I will push my scripts here in a moment
Test_streams.zip (2.32 KB)

The previous script, will create a series of test streams, I have tested one by one using proffesional tools from Tektronix, Intel, and other to validate they are valid test.

Like this, I want to help you to create scripts that will test each parameter of the GStreamer components, This way we can have a clear picture of memory leaks, DTS/PTS issues, stream complaints etc…

Hi,
For memleak in decoding interlaced h264 stream through nvv4l2decoder, please apply attached patch to gst-v4l2 source, rebuild/replace libgstnvvideo4linux2.so, and try again.
0001-gst-v4l2dec-Fix-MemLeak-Fast-Playback-for-Interlace-.zip (1.68 KB)

Has this issue been fixed/included in jetpack 4.3? I also need to decode some interlaced .ts(h264 byte-stream alignment=nal) and have yet to successfully accomplish this.

Hi,
The fix is not in Jetpack 4.3. Please download gst-v4l2 source code to apply the patch and build libgstnvvideo4linux2.so
https://developer.nvidia.com/embedded/dlc/r32-3-1_Release_v1.0/Sources/T210/public_sources.tbz2