Nvv4l2h265enc crash problem

yes,we release pipeline source and use new resolution to start pipeline,repeated;
just like this topic Camera liveview cannot show up again after repeated open/close the liveview for several times - #23 by DaneLLL
but on my test,gst_element_send_event will be locked!!!

And in my another test,delete send eos,it repeated 14276 times and crashed,locked on gst_pad_stop_task
steps:
1 gst_element_set_state(rts->pipeline, GST_STATE_NULL);
2 gst_object_unref(rts->pipeline);
3 g_usleep(1000000);
4 reinitialize pipeline

Hi,
We would need your help to share a test sample based on either
Camera liveview cannot show up again after repeated open/close the liveview for several times - #19 by DaneLLL
or
Camera liveview cannot show up again after repeated open/close the liveview for several times - #23 by DaneLLL

So that we can reproduce the failure and check.

R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t186ref, EABI: aarch64, DATE: Fri Jan 15 23:21:05 UTC 2021

gst-20220112.tar.gz (286.6 KB)

on my test,it also some memory leak, repeated 500time,memory will add 3%[top cmd],just like this top[Using nvv4l2h264enc encoder memory increases when releasing pipeline and setting up a new one - #19 by DaneLLL], how about the 32.5 version ?

Hi,
The fix for the topic is in nvvidconv plugin and r32.5 contains the fix. Fir this issue, please try
Memory Leak (Alloc/free mismatch) in Tegra multimedia API (encoder) - #6 by DaneLLL

ok,i will try that topic fix so file!
what about the lowlevellock issue?

i test ,it also occure memory leak !
i test on 3 tx2;one repeat restart gstreamer pipeline 76449 times,memory from 1% to 37.8%;another repeat restart gstreamer pipeline 81399 times,memory from 1% to 25.4%;the third repeat restart gstreamer pipeline 55893 times,memory from 1% to 28.4%;
the memory lead can test on my attach[gst-20220112.tar.gz on the above reply],i change nvv4l2h265enc,omxh265enc,all test this memory leak.

Hi,
Sorry I did not put correct link in the comment

I have correct it and please give it a try.

no change on memory leak

Hi,
This looks to be another issue. Please check if it helps by dropping cache:
How to Clear RAM Memory Cache, Buffer, and Swap on Linux

And if you stay on r32.5.0, please follow the steps to rename libv4l2_nvargus.so. Some issues are reported with the prebuilt lib and please rename it so that is is not loaded.

yes,always the same test code from the topic,just
modify the pipeline element,it will occure crash and memory leak problem[Camera liveview cannot show up again after repeated open/close the liveview for several times - #23 by DaneLLL]
libv4l2_nvargus have renamed;

huilian@huilian-desktop:~/xxw/rtp-send$ head -n 1 /etc/nv_tegra_release

R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t186ref, EABI: aarch64, DATE: Fri Jan 15 23:21:05 UTC 2021

huilian@huilian-desktop:~/xxw/rtp-send$ sudo find / -name libv4l2_nvargus*
[sudo] password for huilian:
find: ‘/run/user/1000/gvfs’: Permission denied
/usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargusbak.so
/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargusbak1.so
find: ‘/proc/7481/task/7481/net’: Invalid argument
find: ‘/proc/7481/net’: Invalid argument
huilian@huilian-desktop:~/xxw/rtp-send$

test code & gcc
huilian@huilian-desktop:~/xxw/test1$ ls
a.out test.cpp
huilian@huilian-desktop:~/xxw/test1$ cat test.cpp
#include
#include
#include
#include <gst/gst.h>
using namespace std;

#define USE(x) ((void)(x))

static GstPipeline *gst_pipeline = nullptr;
static string launch_string;

GstClockTime usec = 1000000;

int main(int argc, char** argv) {
USE(argc);
USE(argv);
gst_init (&argc, &argv);

GMainLoop *main_loop;
main_loop = g_main_loop_new (NULL, FALSE);
ostringstream launch_stream;
GstBus* bus=NULL;
gboolean  res;

//mem leak test1
#if 1
launch_stream
<< "videotestsrc is-live=1 ! "
<< "video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! "
<< "nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! "
<< “nvv4l2h265enc ! fakesink sync=false async=false”;
#endif
//mem leak test2
#if 0
launch_stream
<< "videotestsrc is-live=1 ! "
<< "video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! "
<< "nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! "
<< “omxh265enc ! fakesink sync=false async=false”;
#endif
//mem leak test3
#if 0
launch_stream
<< "videotestsrc is-live=1 ! "
<< "video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! "
<< "nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! "
<< “fakesink sync=false async=false”;
#endif

launch_string = launch_stream.str();

g_print("Using launch string: %s\n", launch_string.c_str());

GError *error = nullptr;
for(guint i=0; i<20000; i++) {
    printf("loop= %d\n", i);
    gst_pipeline  = (GstPipeline*) gst_parse_launch(launch_string.c_str(), &error);

    if (gst_pipeline == nullptr) {
        printf("Failed to parse launch: %s\n", error->message);
        return -1;
    }
    if(error) 
    {
        printf("Parse error[%d]\n",i);
        g_error_free(error);
    }

    printf("Set playing[%d]\n",i);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_PLAYING); 
    g_usleep(3*usec);
    printf("Send Eos[%d]\n",i);
    res=gst_element_send_event ((GstElement*)gst_pipeline, gst_event_new_eos ());
    if(!res)
    {
        printf("eos signal send failed\n");
    }
    else
    {
        printf("eos signal send success\n"); 
    }
    printf("get bus[%d]\n",i);
    bus = gst_pipeline_get_bus(GST_PIPELINE(gst_pipeline));
    printf("bus poll start[%d]\n",i);
    gst_bus_poll(bus, GST_MESSAGE_EOS, GST_CLOCK_TIME_NONE);
    printf("bus poll end &Set NULL[%d]\n",i);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);
    printf("Set unref[%d]\n",i);
    gst_object_unref(GST_OBJECT(gst_pipeline));
gst_object_unref (bus);
gst_pipeline=NULL;
bus=NULL;
    g_usleep(1*usec);
fflush(stdout);
}
g_main_loop_unref(main_loop);

g_print("going to exit \n");
return 0;

}
huilian@huilian-desktop:~/xxw/test1$ g++ -g test.cpp pkg-config --cflags --libs gstreamer-1.0
huilian@huilian-desktop:~/xxw/test1$

Hi,
We apply this and run below script on r32.6.1/TX2 overnight. Don’t observe the issue:

#!/usr/bin/env python3

import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject, GLib

pipeline = None
bus = None
message = None

# initialize GStreamer
Gst.init(None)

for i in range(1, 65535):
    print("loop =",i," ")
    # build the pipeline
    pipeline = Gst.parse_launch(
        "videotestsrc is-live=1 ! video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! nvv4l2h265enc ! fakesink "
    )

    # start playing
    print("Switch to PLAYING state")
    pipeline.set_state(Gst.State.PLAYING)

    time.sleep(5)
    print("Send EoS")
    Gst.Element.send_event(pipeline, Gst.Event.new_eos())
    # wait until EOS or error
    bus = pipeline.get_bus()
    msg = bus.timed_pop_filtered(
        Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS)

    # free resources
    print("Switch to NULL state")
    pipeline.set_state(Gst.State.NULL)
    time.sleep(1)

The memory usage looks steady:
[start: PID 9960]
image

[13381 times]
image

Could you try it on r32.5.0? To check if the issue is specific to the version.

i test overnight,mem from 0.8% to 3.9% ,7795 times



i add GST_TRACE env, and run 50 times and end over with gst_deinit()
it printf leaks detected of GstCaps of encoder
0:03:33.169063483 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f9808a320, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);
0:03:33.169080602 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x55a16db230, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);
0:03:33.169099001 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f9405ea30, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);
0:03:33.169116377 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x55a1706c00, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);

** (test5:30128): WARNING **: 12:25:00.173: Leaks detected


i add GST_TRACE env and over with gst_deinit,
nvv4l2h264enc,nvv4h265enc will appear leaks about GstCaps;but omxh265enc have not leaks !!!

Hi,
Do you upgrade to r32.6.1 and do the test? Since we don’t see significant leak on r32.6.1, you may consider to upgrade to the version. We are deprecating omx plugins, but if the plugins work fine in your use-case, you may still consider to use it.

  1. I did not upgrade to r32.6.1,just test on r32.5.0;

  2. omxh265enc just for test,my project now use nvv4l2h265enc on r32.5.0,i do not want to update too often.

3.review the gst-v4l2 source,the gstv4l2h265enc.c use gst_static_caps_get(&src_template_caps) and not unref

I downloaded and checked the source code of ‘nvv4l2h265enc’ from:
https://developer.download.nvidia.cn/embedded/L4T/r32_Release_v5.1/r32_Release_v5.1/sources/T186/public_sources.tbz2

In file ‘gstv4l2h265enc.c’, line 391~line 404:

gboolean
gst_v4l2_is_h265_enc (GstCaps * sink_caps, GstCaps * src_caps)
{
return gst_v4l2_is_video_enc (sink_caps, src_caps,
gst_static_caps_get (&src_template_caps));
}

void
gst_v4l2_h265_enc_register (GstPlugin * plugin, const gchar * basename,
const gchar * device_path, GstCaps * sink_caps, GstCaps * src_caps)
{
gst_v4l2_video_enc_register (plugin, GST_TYPE_V4L2_H265_ENC,
“h265”, basename, device_path, sink_caps,
gst_static_caps_get (&src_template_caps), src_caps);
}

These two APIs both call 'gst_static_caps_get() '.
And from:

/**

  • gst_static_caps_get:
  • @static_caps: the #GstStaticCaps to convert
  • Converts a #GstStaticCaps to a #GstCaps.
  • Returns: (transfer full) (nullable): a pointer to the #GstCaps. Since the
  • core holds an additional ref to the returned caps, use
    
  • gst_caps_make_writable() on the returned caps to modify it.
    

*/
GstCaps *
gst_static_caps_get (GstStaticCaps * static_caps)
{
GstCaps **caps;

g_return_val_if_fail (static_caps != NULL, NULL);

caps = &static_caps->caps;

/* refcount is 0 when we need to convert */
if (G_UNLIKELY (*caps == NULL)) {
const char *string;

G_LOCK (static_caps_lock);
/* check if other thread already updated */
if (G_UNLIKELY (*caps != NULL))
  goto done;

string = static_caps->string;

if (G_UNLIKELY (string == NULL))
  goto no_string;

*caps = gst_caps_from_string (string);

/* convert to string */
if (G_UNLIKELY (*caps == NULL)) {
  g_critical ("Could not convert static caps \"%s\"", string);
  goto done;
}

/* Caps generated from static caps are usually leaked */
GST_MINI_OBJECT_FLAG_SET (*caps, GST_MINI_OBJECT_FLAG_MAY_BE_LEAKED);

GST_CAT_TRACE (GST_CAT_CAPS, "created %p from string %s", static_caps,
    string);

done:
G_UNLOCK (static_caps_lock);
}
/* ref the caps, makes it not writable */
if (G_LIKELY (*caps != NULL))
gst_caps_ref (*caps);

return *caps;

/* ERRORS */
no_string:
{
G_UNLOCK (static_caps_lock);
g_warning (“static caps %p string is NULL”, static_caps);
return NULL;
}
}

Seems that ‘gst_static_caps_get()’ will return a GstCaps object with refcount >= 1.
And this GstCaps object should be ‘unref’-ed after use, in order to release the object.
However, this release is missed in both ‘gst_v4l2_is_h265_enc’ and ‘gst_v4l2_h265_enc_register’, which may possibly cause the memory leak,
as indicated by :

0:03:33.169063483 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f9808a320, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);

hi,danelll,what about the isuue