Decoding H264 1920x1080i50 (Interlace video) makes the decoding engine to slowdown dramatically over time, tegrastats reports an increase of memory of around 34.7MB per minute decoded.
Intial load: from 1412/3957MB to 1436MB, decoding around 515=6 frames per second (executable load 24MB, 220 frames decoded)
The memory leak, has something to do with the fame size allocated, When decoding 720x576i50 the leak is about 9.2MB per minute of video decoded Performance also degraded dramatically.
checked every component of the pipeline (no leaks on filesrc nor h264parse
nvv4l2decoder sometimes crashes after 2.2min of HD decoding (Used different content) and 6:50min on SD PAL, Leak on nvvidconv is smaller than omxh264dec, BUT performance degradation is similar (Went down from 480fps to 126fps on by 00:20:00 of the stream and to 64fps after 00:40:00)
Used a variety of 1080i50 professional content encoded with Harmonics/Elemental/enViVo and even x264
** This problem is not happening on MPEG2, and for completeness the Jetson Nano decoded 239fps MPEG2 1920x1080 frames
** This problem is not happening on H264 Progressive video, and for completeness the Jetson Nano decoded 284fps H264 1920x1080 frames
I read about the deprecation of omx Plug ins. And as you can see on the report, I also did also the test on nvv4l2decoder, And yes the leak is lower (20X lower), but not something it can be used for a long time. Apart from the leak, the main problem on both elements, is the slowing down of decoding performance, Check the fps performance I posted, you will be able to see the degradation to levels where HW acceleration defeats the purpose to the point where is even more convenient to use an H264 SW decoder using the 4 CPUs.
BTW: I used tegrastats to build up the report table. Also the critical degradation happens around 15 minutes of decoding on HD
Hi Rudoplh,
Please execute ‘sudo jetson_clcoks’,enable following property, and try again:
enable-max-performance: Set to enable max performance
flags: readable, writable
Boolean. Default: false
Please also check thermal in tegrastats.
Hi RaviKiranK,
Your comment is different from this topic, although the streamer pipeline looks good. You may make a new post, if you still hit issues in running gstreamer + OpenCV
I ran the same test using the same video content (But encoded as progressive this time) and the system doesn’t show this problem.
Memory footprint 363MB before test. 437MB while running and grow to 459MB after 269,955 Progressive Frames decoded (3 hours of video decoded) and relaxed to 435MB after the program ended execution. (8MB leak per hour of 1080p video decoded.
Decoder performance 257Fps and 280Fps with jetson_clocks
I couldn’t identify the source of the problem, however I could confirm that it exists on progressive inputs as well.
Not to bear any hard feelings, but I find it ironic that on my topic I was told to use omx and here OP was told otherwise…
On a side note, there was recently an update to gstreamer 1.14.4 (previous was 1.14.1) however nvv4l2 codecs all report version 1.14.0 so I doubt something changed there…
Hi 11wallace11,
Your issue is investigated and shall be fixed in next r32.2. If you are going to production based on r32.1 and cannot wait for r32.2, please let us know more about your project.
The issue I’m reporting is on the decoder side. The encoder (As a element) is working fine for me excepting for a bug on the b frames on H264 that I’m going to post in a few minutes. My testings are 3 days of content long and is stable (no memory leaks and reliable)
What you might have experienced is that Nvidia decided to use a low level media engine such as GSTreamer. GStreamr have a prototype tool called gst-launch, this is tool doesn’t handle many of the stuff that makes a commercial product reliable, such drop of connections and recovery, glitches on the stream, lack of standard complains by some vendors, buffer handling, etc. instead you need to build your own app using this high-level media abstraction. or like me, drop all this open source stuff and build your own SW using C++ and the NMAP components. (NMAP as a NVIDIA Multimedia Applications)
*full disclosure, I don’t work for Nvidia and I hope it helps…
Thanks for your kind offer and pointing me in the right direction to look for the source code.
I’m not in a hurry and can definitely wait for the next version to come out :)
However, if there’s a public beta or beta-testing program I’ll be more than happy to participate and issue detailed bug reports.
I’ve been involved in the “media codecs” scene for more than 10 years now, and while I probably have the basic knowledge necessary to build something that’d work, I feel that it’s not worth my time to start working on a new project for a platform that’s being dominated by Nvidia…
That being said, if Nvidia had a proper git repository somewhere and accepted participation of developers not from the company, it may be a good incentive for me (and probably other developers as well) to participate.
To sum up my stance since it’s probably still not so clear at point, I prefer an open repository that everyone worked on (or forked) instead of the current “open source” form of “here, take the source code and develop something for yourself, and yourself only”.
I’m terribly sorry for deviating so much from the original topic. Feel free to ignore my comment :)
The problem on the decoding remains, In fact it went much worst than the status of r32.1.
r32.2 and r32.2.1 which are the same gst-nvvideo4linux2_src code, made the decoding of H264 on the Jetson Nano totally useless. It has being almost 4 months and the problem rather than get fixed went much worst and we have commitments to fulfill.
omxh264dec there is a memory leak, but at least we know that the Jetson is able to play H.264 interlace properly. The memory leak appears to be because the videodecoder Sub-class should implement drain()
The new nvv4l2decoder is a disaster. Rather than fix bugs, It looks like nVidia decided to add more features like a new buffer method and the support of DIVx, something that probably nobody uses since more than 10 years ago (Probably pirates)
The new problem is that the decoding of H264i it is always running like in sync=false, in other words it is playing super fast and without timing control, or the timing of the output is in fields?. In addition there is some variations of H264, that doesn’t play at all (I can share professional stream examples), in other samples it looks like the chroma planes are gone. This problems are not happening on omxh264dec. The caps reported by the decoder are not right they used to report progressive output but now is “mixed.” This can be fixed by putting back a pice of code that was moved. See the picture.
Moving forward: As developers, we are relying on nVidia as a product, as developers we cannot move out of the BETA stage and we need Nvidia to understand that this is not right and is costing us business.
I like @11wallece11 ide to put the gst-nvvideo4linux2_src repo online, maybe at nvidia · GitLab or NVIDIA Corporation · GitHub, so we can help with testing and development in a more agile way.
Part of the memory leak, the Cache part can be fixed by doing
sysctl vm.drop_caches=3
nvv4l2decoder: Have other broken parts, Per example the disableDB=true is producing problems too. The new bufferAPI feature, seems to have no effect into the pipe.
I will he happy to help by making a report and a full test of your software. BUT to be honest, I’m disturbed by how nVidia is releasing software without proper testing or quality control, or adding a note of the know issues and potential fixing schedule.
p.s. I can also put the repo into GitHub, as long nVidia is committed to put the daily changes on that repo to allow other developers help you.
Hi Rudoplh,
We have debugged further and tried similar set up with odroid board (which uses gst-v4l2 open source plug-ins) and find similar observations of increasing vmRSS during playback of long duration interlaced streams. And we are checking to file one bug to the gstreamer-plugins-good community as this seems to be a open source related issue.
For the issue about setting ‘disableDB=true’, if it is a separate issue from decoding interlaced streams, please start a new post for it. Thanks.
The memory leak of the omxh264dec is probably related to a warning that gstreamer is reporting s a FIX ME! because the videodecoder Sub-class should implement drain(). I know that someone at nVidia decided no abandon this to focus on nvv4l2decoder. In this case nVidia I believe it will be very helpful for ALL of us that nVidia release the repo into gitLab or GitHub, so we can track the changes made and continue with this effort, as it looks like is easier to fix that the new nvv4l2decoder
nvv4l2decoder. Is working “more or less” ok on 32.1, it is seriously BROKEN on 32.2 and 32.1. This cannot be gstreamer fault.
PLEASE, I know that your group at nVidia is a very small, BUT this group cannot just dismiss this problems and leave your customers in the limbo. Each release posted to the public requires serious testing before the code is release it. It has being months, and we have made commitments and serious investments believing that nVidia released a PRODUCT.
PLEASE, PLEASE, check this issue, It is not gstreamer, the code r.32.2 and r32.2.1 have issues as I pointed, and this code is made by nVidia, no gStreamer.
BTW: The issue is not on long durations streams, in less than few seconds, you will see the memory leak issues.
