Possible memory leak in nvvidconv gstreamer plugin or decoder?

Hi,

I can see a strange behaviour with this simple code reading a RTSP stream from an IP camera (hikvision one at 1920x1080).
Memory of my program starts at 2.2% of the total when running for the first time the stream, and then each time I stop and I start the stream, the memory is increasing (GPU memory is stable).
It seems to be more stable when reaching 7.1% of the total but then seems to increase less quick than at the beginning.
This is very strange, and I cannot afford this kind of “leak” (if it’s a leak, maybe on my code), because this is for 1 stream, but we plan to receive up to 8 streams.

The program is running the stream for 3 sec, stops it for 3 secs, in an infinite loop until Ctrl+C is pressed.
I check memory usage of the program using htop watching at the percent of the total, and GPU usage using “cat /proc/meminfo | grep NvMapMemUsed” @dusty_nv told me on another thread.
The code is a bit ugly (using global, debug prints, etc) but it’s to show the problem.

#include <gst/gst.h>
#include <gst/app/gstappsink.h>
#include <gstreamer-1.0/gst/rtsp/gstrtsptransport.h>
#include <gio/gio.h>
#include <iostream>
#include <cassert>

GMainLoop* g_loop = nullptr;

void on_eos(GstAppSink* app_sink, gpointer user_data)
{
    std::cout << "on_eos" << std::endl;
}

GstFlowReturn on_new_preroll(GstAppSink* app_sink, gpointer user_data)
{
    static std::size_t i = 0;
    std::cout << "on_new_preroll " << i++ << std::endl;

    GstSample* sample = gst_app_sink_pull_preroll(app_sink);
    if (sample != nullptr)
    {
        GstCaps* caps = gst_sample_get_caps(sample);
        if (caps != nullptr)
        {
            GstStructure* structure = gst_caps_get_structure(caps, 0);
            if (structure != nullptr)
            {
                int width;
                gst_structure_get_int(structure, "width", &width);

                int height;
                gst_structure_get_int(structure, "height", &height);

                std::cout << "width: " << width << ", height: " << height << std::endl;

                const gchar* name = gst_structure_get_name(structure);
                if (strcmp(name, "video/x-raw") == 0)
                {
                    const gchar* format = gst_structure_get_string(structure, "format");
                    if (format != nullptr)
                    {
                        std::cout << "format: " << format << std::endl;
                    }
                }

                GstBuffer* buffer = gst_sample_get_buffer(sample);
                if (buffer != nullptr)
                {
                    GstMapInfo map;
                    if (gst_buffer_map(buffer, &map, GST_MAP_READ) == TRUE)
                    {
                        std::cout << "size: " << map.size << std::endl;
                        gst_buffer_unmap(buffer, &map);
                    }
                }
            }
        }
        gst_sample_unref(sample);
    }
    return GST_FLOW_OK;
}

GstFlowReturn on_new_sample(GstAppSink* app_sink, gpointer user_data)
{
    static std::size_t i = 0;
    std::cout << "on_new_sample " << i++ << std::endl;

    GstSample* sample = gst_app_sink_pull_sample(app_sink);
    if (sample != nullptr)
    {
        GstCaps* caps = gst_sample_get_caps(sample);
        if (caps != nullptr)
        {
            GstStructure* structure = gst_caps_get_structure(caps, 0);
            if (structure != nullptr)
            {
                int width;
                gst_structure_get_int(structure, "width", &width);

                int height;
                gst_structure_get_int(structure, "height", &height);

                std::cout << "width: " << width << ", height: " << height << std::endl;

                const gchar* name = gst_structure_get_name(structure);
                if (strcmp(name, "video/x-raw") == 0)
                {
                    const gchar* format = gst_structure_get_string(structure, "format");
                    if (format != nullptr)
                    {
                        std::cout << "format: " << format << std::endl;
                    }
                }

                GstBuffer* buffer = gst_sample_get_buffer(sample);
                if (buffer != nullptr)
                {
                    GstMapInfo map;
                    if (gst_buffer_map(buffer, &map, GST_MAP_READ) == TRUE)
                    {
                        std::cout << "size: " << map.size << std::endl;
                        gst_buffer_unmap(buffer, &map);
                    }
                }
            }
        }
        gst_sample_unref(sample);
    }
    return GST_FLOW_OK;
}

void on_source_setup(GstElement* /*bin*/, GstElement* source, gpointer /*data*/)
{
    const gchar* type_name = g_type_name(G_OBJECT_TYPE(source));
    if (strcmp(type_name, "GstRTSPSrc") == 0)
    {
        g_object_set(G_OBJECT(source), "protocols", GST_RTSP_LOWER_TRANS_TCP, NULL);
        g_object_set(G_OBJECT(source), "latency", 300, NULL);
        g_object_set(G_OBJECT(source), "user-id", "admin", NULL);
        g_object_set(G_OBJECT(source), "user-pw", "MYPASSWORD", NULL);
    }
}

gboolean print_cap(GstCapsFeatures* features, GstStructure* structure, gpointer user_data)
{
    gchar* s = gst_structure_to_string(structure);
    g_print("%s\n", s);
    g_free(s);
    return TRUE;
}

void on_source_pad_added(GstElement* element, GstPad* pad, gpointer data)
{
    g_print("Dynamic pad created, linking source -> convert\n");
    GstElement* convert = reinterpret_cast<GstElement*>(data);
    GstCaps* caps = gst_pad_get_current_caps(pad);
    gst_caps_foreach(caps, print_cap, nullptr);
    gst_caps_unref(caps);
    GstPad* sinkpad = gst_element_get_static_pad(convert, "sink");
    gst_pad_link(pad, sinkpad);
    gst_object_unref(sinkpad);
}

void print_gst_error(GstMessage* msg)
{
    gchar* debug;
    GError* error;
    gst_message_parse_error(msg, &error, &debug);
    g_printerr("Error: %s, Debug: %s\n", error->message, debug);
    g_free(debug);
    g_error_free(error);
}

gboolean bus_call(GstBus* bus, GstMessage* msg, gpointer data)
{
    switch (GST_MESSAGE_TYPE(msg))
    {
    case GST_MESSAGE_EOS:
        std::cout << "bus_call: GST_MESSAGE_EOS" << std::endl;
        break;
    case GST_MESSAGE_ERROR:
        std::cout << "bus_call: GST_MESSAGE_ERROR" << std::endl;
        print_gst_error(msg);
        break;
    default:
        break;
    }

    return FALSE;
}

void handle_signal(int sig)
{
    if (sig == SIGINT)
    {
        std::cout << "SIGINT" << std::endl;
        g_main_loop_quit(g_loop);
        signal(SIGINT, SIG_DFL);
    }
}

GstElement* g_pipeline = nullptr;

struct args
{
    int argc;
    char** argv;
};

gboolean stop_stream_callback(gpointer user_data);
gboolean start_stream_callback(gpointer user_data);

bool start(args* args)
{
    std::cout << "start()" << std::endl;

    //gst_init(&args->argc, &args->argv);
    
    GstElement* uri_decode_bin = gst_element_factory_make("uridecodebin", "uridecodebin");
    GstElement* convert = gst_element_factory_make("nvvidconv", "convert");
    GstElement* sink = gst_element_factory_make("appsink", "sink");

    if (uri_decode_bin != nullptr &&
        convert != nullptr &&
        sink != nullptr)
    {
        GstAppSink* app_sink = GST_APP_SINK(sink);

        // Configure source
        g_object_set(G_OBJECT(uri_decode_bin), "uri", "rtsp://192.168.10.52/Streaming/Channels/1", NULL);

        // Configure sink
        gst_app_sink_set_emit_signals(app_sink, TRUE);
        gst_app_sink_set_drop(app_sink, TRUE);
        gst_app_sink_set_max_buffers(app_sink, 1);
        //g_object_set(sink, "drop", TRUE, NULL);
        //g_object_set(sink, "max-buffers", 1, NULL);
        GstCaps* caps = gst_caps_new_simple("video/x-raw", "format", G_TYPE_STRING, "BGRx", NULL);
        gst_app_sink_set_caps(app_sink, caps);
        gst_caps_unref(caps);

        // Pìpeline setup
        g_pipeline = gst_pipeline_new("pipeline");
        assert(g_pipeline != nullptr);

        gst_bin_add_many(GST_BIN(g_pipeline), uri_decode_bin, convert, sink, NULL);
        gst_element_link_many(convert, sink, NULL);

        //g_signal_connect(g_pipeline, "deep-notify", G_CALLBACK(gst_object_default_deep_notify), NULL);
        g_signal_connect(uri_decode_bin, "pad-added", G_CALLBACK(on_source_pad_added), convert);
        // Configure source
        g_signal_connect(uri_decode_bin, "source-setup", G_CALLBACK(on_source_setup), NULL);

        // appsink signals
        g_signal_connect(app_sink, "new-preroll", G_CALLBACK(on_new_preroll), NULL);
        g_signal_connect(app_sink, "new-sample", G_CALLBACK(on_new_sample), NULL);
        g_signal_connect(app_sink, "eos", G_CALLBACK(on_eos), NULL);

        GstBus* bus = gst_pipeline_get_bus(GST_PIPELINE(g_pipeline));
        gst_bus_add_watch(bus, bus_call, NULL);
        gst_object_unref(bus);

        gst_element_set_state(g_pipeline, GST_STATE_PLAYING);

        return true;
    }
    else
    {
        gst_object_unref(GST_OBJECT(g_pipeline));
        gst_object_unref(GST_OBJECT(uri_decode_bin));
        gst_object_unref(GST_OBJECT(convert));
        gst_object_unref(GST_OBJECT(sink));

        return false;
    }
}

void stop()
{
    std::cout << "stop()" << std::endl;
    if (g_pipeline != nullptr)
    {
        gst_element_set_state(g_pipeline, GST_STATE_NULL);
        gst_object_unref(GST_OBJECT(g_pipeline));
        g_pipeline = nullptr;
    }
    //gst_deinit();
}

gboolean stop_stream_callback(gpointer user_data)
{
    stop();
    g_timeout_add_seconds(4, start_stream_callback, user_data);
    return FALSE;
}

gboolean start_stream_callback(gpointer user_data)
{
    start(reinterpret_cast<args*>(user_data));
    g_timeout_add_seconds(4, stop_stream_callback, user_data);
    return FALSE;
}

int main(int argc, char* argv[])
{
    gst_init(&argc, &argv);

    args args;
    args.argc = argc;
    args.argv = argv;
    if (start(&args))
    {
        g_timeout_add_seconds(4, stop_stream_callback, &args);

        struct sigaction signal_handler;
        signal_handler.sa_handler = handle_signal;
        sigemptyset(&signal_handler.sa_mask);
        signal_handler.sa_flags = 0;
        sigaction(SIGINT, &signal_handler, nullptr);

        g_loop = g_main_loop_new(NULL, FALSE);
        g_main_loop_run(g_loop);

        stop();

        std::cout << "gst_deinit()" << std::endl;
        gst_deinit();

        g_main_loop_unref(g_loop);

        return 0;
    }
    else
    {
        return 1;
    }
}

EDIT:
After more investigation, using
valgrind --tool=massif --time-unit=ms --detailed-freq=1 --threshold=0.5 --suppressions=/usr/share/glib-2.0/valgrind/glib.supp --depth=30 --alloc-fn=g_malloc --alloc-fn=g_realloc --alloc-fn=g_try_malloc --alloc-fn=g_malloc0 --alloc-fn=g_mem_chunk_alloc myprogram

I get this result in massif-visualizer, maybe the problem is not related to nvvidconv… maybe someone here has had this problem?

EDIT2:
I have more information! I installed the dbg packages of base/ugly/good/bad on the jetson, and run valgrind again, now it gives me this:

It seems to be when the gstreamer receives SPS and PPS h264 NALs…

EDIT3:
If I run

GST_TRACERS="leaks" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 uridecodebin uri=rtsp://admin:MYPASSWORD@192.168.10.52/Streaming/Channels/1 ! nvvidconv ! video/x-raw, format=BGRx ! fakesink -v

I get the following:

0:00:02.024830313 32299   0x55c1048870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f2c079450, description=(string)EMPTY, ref-count=(uint)1, trace=(string);
0:00:02.024909690 32299   0x55c1048870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f2c079e80, description=(string)video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1, ref-count=(uint)1, trace=(string);

And if I run:

GST_TRACERS="leaks" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 uridecodebin uri=rtsp://admin:MYPASSWORD@192.168.10.52/Streaming/Channels/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12 ! fakesink -v

I get the following:

0:00:01.384970454  1684   0x5594793870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f640b2140, description=(string)video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1, ref-count=(uint)1, trace=(string);

Does this mean there is really a leak in nvvidconv?

Hi,
For information, please share the release version you use( $ head -1 /etc/nv_tegra_release ).

Here is a sample similar to your usecase:


Please give it a try and see if you observe the same.

R32 (release), REVISION: 4.2, GCID: 20074772, BOARD: t210ref, EABI: aarch64, DATE: Thu Apr 9 01:22:12 UTC 2020

I am using jetpack 4.4 DP but it happened with 4.3 too.
I’ll have a look at the link you sent me.
Thanks

Ok, we came to the conclusion that it seems to be memory fragmentation, process memory usage seems to stabilize after some time.

However I think there’s a leak in nvv4l2decoder, or maybe not, I’m not sure my experiment is correct, what do you think about this:

GST_TRACERS="leaks" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 rtspsrc location=rtsp://admin:PWD@192.168.10.52/Streaming/Channels/1 latency=300 ! rtph264depay ! nvv4l2decoder ! nvvidconv ! "video/x-raw, format=BGRx" ! fakesink -v

0:00:06.472014345  8178   0x5568a5a870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f2c003190, description=(string)EMPTY, ref-count=(uint)1, trace=(string);
0:00:06.472072263  8178   0x5568a5a870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f2c003850, description=(string)EMPTY, ref-count=(uint)1, trace=(string);
0:00:06.472956717  8178   0x5568a5a870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f34003320, description=(string)video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)0/1, ref-count=(uint)1, trace=(string);
0:00:06.473070783  8178   0x5568a5a870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f2c003800, description=(string)EMPTY, ref-count=(uint)1, trace=(string);
** (gst-launch-1.0:8178): WARNING **: 09:58:52.111: Leaks detected

If I run with omx decoder:

GST_TRACERS="leaks" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 rtspsrc location=rtsp://admin:PWD@192.168.10.52/Streaming/Channels/1 latency=300 ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! "video/x-raw, format=BGRx" ! fakesink -v

This Gstreamer leak appears in my program too if I use nvv4l2decoder, but not if I use omxh264dec.

I use BGRx for the fakesink because it’s what I need to use (for now) in my program.

Regards