Nvvidconv memory leak suspected

Hello,

Hardware: Jetson Xavier
Software

  • L4T 32.6.1
  • GStreamer Core Library version 1.14.5
  • nvvidconv: 1.2.3

We are using the nvvidconv plugin in a pipeline that performs ROI transforms, scaling of raw images and jpeg encoding.

When using the nvvidconv plugin, we’re noticing that it leaks GPU memory consistently and very quickly when used with an upstream videocrop plugin that changes dimensions of the input image. We suspect the nvvidconv element leaks GPU memory everytime the crop dimensions are changed.

Here is a minimal Python script to reproduce the issue:

import asyncio
from concurrent.futures import ThreadPoolExecutor
import random

import gi
gi.require_version('Gst', '1.0')

from gi.repository import GLib, Gst
Gst.init(None)

pipeline = Gst.parse_launch("videotestsrc ! videocrop name=crop top=16 left=16 right=16 bottom=16 ! video/x-raw, framerate=10/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12, width=224, height=224, framerate=10/1 ! autovideosink")
crop = pipeline.get_by_name('crop')
pipeline.set_state(Gst.State.PLAYING)

main_loop = GLib.MainLoop()
loop = asyncio.get_event_loop()
task = loop.run_in_executor(ThreadPoolExecutor(), main_loop.run)

# Switch videocrop upstream from nvvidconv leads to leakage
async def switch_scale(crop):
    for _ in range(1000):
        x = random.randint(0, 100)
        crop.set_property('top', x)
        crop.set_property('top', x)
        crop.set_property('top', x)
        crop.set_property('top', x)
        await asyncio.sleep(0.1)
        print("frame")

# Option 1: Leaks GPU memory like crazy when performing crops
loop.run_until_complete(asyncio.gather(task,switch_scale(crop)))

# Option 2: Does not leak GPU memory
#loop.run_until_complete(asyncio.gather(task,))

Where Option 1 will leak (about 4MB/s) when monitoring jtop’s GPU MEM output, and Option 2 will not.

I’ve found these issues that might be related:

Is it possible to get an newer version of the nvvidconv plugin that may have had this memory leak addressed for Jetpack 4.6 that I can use with my version of Gstreamer?

Here are two screenshots of jtop indicating GPU usage, taken approximately 2 minutes apart:

First:

About two minutes later:

As can be seen from the images, the GPU memory allocation has grown from 11.4MB to 181MB in a short period of time. It will continue growing.

Thanks much.

Hi,
Dynamically changing crop region is not supported in default nvvidconv plugin. We would suggest use NvBuffer APIs for the use-case. You can call NvBufferTransform() to crop the frames.

For using gstreamer, the nvvidconv plugin is open source. You may look at the source code and customize it to support dynamic crop region.

Thanks for your response.

I do not wish to dynamically change the crop region using the nvvidconv tool, that was not the intent of this post.
I am pointing out that there is a significant NVMM memory leak in the nvvidconv gst plugin when changing the dimensions of the buffer on the nvvidconv sink pad.

My expectation is that the plugin handles this like any other Gstreamer plugin and auto negotiates the new buffer size.

In actuality, while the nvvidconv operates as expected as far as auto negotiating the new buffer size and the pipeline continues playing, the plugin leaks a lot of NVMM memory any time the input buffer size and dimensions are changed. This leads me to believe that the plugin does not properly deallocate memory when it needs to reallocate a new buffer size for the stream.

Please point me to the repository that has the open source gst plugin code for nvvidconv. I’d like to see if I can compile the latest against my gstreamer build on Jetpack 4.6 - perhaps this issue has been remedied already.

Please also advise on whether the memory leak is known and whether a fix already exists. I am using an old version of the plugin and if there is a way to use a newer build without having to build it myself, I’d much appreciate any information on how to access it.

Hi,
Please download the source code package:
https://developer.nvidia.com/embedded/linux-tegra-r3261

L4T Driver Package (BSP) Sources

And do customization to nvvidconv plugin for the use-case.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.