Nvv4l2h265enc crash problem

[happy886]

I’m able to send rtp streams across the network using nvv4l2h265enc and receive rtp streams in gstreamer pipeline for c program code;
To adapt to the poor situation of the network,the stream resolution will change:[1280x960,640x480,480x360];
but after change resolution test 488 times, nvv4l2h265enc crashed

this is main loop,in order to speed up the test,shortened sleep time;

this is the function code

just as this cmd gst pipeline
gst-launch-1.0 -v rtpbin name=rtpbin v4l2src device=/dev/video1 !
“video/x-raw,format=UYVY,width=1280,height=960,framerate=30/1” !
nvvidconv !
“video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1” !
nvv4l2h265enc control-rate=1 EnableTwopassCBR=1 bitrate=512000 idrinterval=30
insert-sps-pps=true num-Ref-Frames=1 slice-header-spacing=300 SliceIntraRefreshInterval=30 !
“video/x-h265,stream-format=(string)byte-stream” ! rtph265pay ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 !
udpsink host=192.168.1.5 port=30002 rtpbin.send_rtcp_src_0 !
udpsink host=192.168.1.5 port=30008
udpsrc port=30005 ! rtpbin.recv_rtcp_sink_0

[DaneLLL]

Hi,
Do you re-initialize the gstreamer pipeline when resolution changes? By default it does not support runtime resolution change in nvv4l2h265enc. When resolution changes, please terminate the original pipeline and re-initialize a new one.

[happy886]

yes,we release pipeline source and use new resolution to start pipeline,repeated;
just like this topic Camera liveview cannot show up again after repeated open/close the liveview for several times - #23 by DaneLLL
but on my test,gst_element_send_event will be locked!!!

[happy886]

And in my another test,delete send eos,it repeated 14276 times and crashed,locked on gst_pad_stop_task
steps:
1 gst_element_set_state(rts->pipeline, GST_STATE_NULL);
2 gst_object_unref(rts->pipeline);
3 g_usleep(1000000);
4 reinitialize pipeline

[DaneLLL]

Hi,
We would need your help to share a test sample based on either
Camera liveview cannot show up again after repeated open/close the liveview for several times - #19 by DaneLLL
or
Camera liveview cannot show up again after repeated open/close the liveview for several times - #23 by DaneLLL

So that we can reproduce the failure and check.

[happy886]

R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t186ref, EABI: aarch64, DATE: Fri Jan 15 23:21:05 UTC 2021

gst-20220112.tar.gz (286.6 KB)

[happy886]

on my test,it also some memory leak, repeated 500time,memory will add 3%[top cmd],just like this top[Using nvv4l2h264enc encoder memory increases when releasing pipeline and setting up a new one - #19 by DaneLLL], how about the 32.5 version ?

[DaneLLL]

Hi,
The fix for the topic is in nvvidconv plugin and r32.5 contains the fix. Fir this issue, please try
Memory Leak (Alloc/free mismatch) in Tegra multimedia API (encoder) - #6 by DaneLLL

[happy886]

ok,i will try that topic fix so file!
what about the lowlevellock issue?

[happy886]

i test ,it also occure memory leak !
i test on 3 tx2;one repeat restart gstreamer pipeline 76449 times,memory from 1% to 37.8%;another repeat restart gstreamer pipeline 81399 times,memory from 1% to 25.4%;the third repeat restart gstreamer pipeline 55893 times,memory from 1% to 28.4%;
the memory lead can test on my attach[gst-20220112.tar.gz on the above reply],i change nvv4l2h265enc,omxh265enc,all test this memory leak.

[DaneLLL]

Hi,
Sorry I did not put correct link in the comment

I have correct it and please give it a try.

[happy886]

no change on memory leak

[DaneLLL]

Hi,
This looks to be another issue. Please check if it helps by dropping cache:
How to Clear RAM Memory Cache, Buffer, and Swap on Linux

And if you stay on r32.5.0, please follow the steps to rename libv4l2_nvargus.so. Some issues are reported with the prebuilt lib and please rename it so that is is not loaded.

[happy886]

yes,always the same test code from the topic,just
modify the pipeline element,it will occure crash and memory leak problem[Camera liveview cannot show up again after repeated open/close the liveview for several times - #23 by DaneLLL]
libv4l2_nvargus have renamed;

huilian@huilian-desktop:~/xxw/rtp-send$ head -n 1 /etc/nv_tegra_release

R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t186ref, EABI: aarch64, DATE: Fri Jan 15 23:21:05 UTC 2021

huilian@huilian-desktop:~/xxw/rtp-send$ sudo find / -name libv4l2_nvargus*
[sudo] password for huilian:
find: ‘/run/user/1000/gvfs’: Permission denied
/usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargusbak.so
/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargusbak1.so
find: ‘/proc/7481/task/7481/net’: Invalid argument
find: ‘/proc/7481/net’: Invalid argument
huilian@huilian-desktop:~/xxw/rtp-send$

test code & gcc
huilian@huilian-desktop:~/xxw/test1$ ls
a.out test.cpp
huilian@huilian-desktop:~/xxw/test1$ cat test.cpp
include
include
include
include <gst/gst.h>
using namespace std;

define USE(x) ((void)(x))

static GstPipeline *gst_pipeline = nullptr;
static string launch_string;

GstClockTime usec = 1000000;

int main(int argc, char** argv) {
USE(argc);
USE(argv);
gst_init (&argc, &argv);

GMainLoop *main_loop;
main_loop = g_main_loop_new (NULL, FALSE);
ostringstream launch_stream;
GstBus* bus=NULL;
gboolean  res;

//mem leak test1
#if 1
launch_stream
<< "videotestsrc is-live=1 ! "
<< "video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! "
<< "nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! "
<< “nvv4l2h265enc ! fakesink sync=false async=false”;
endif
//mem leak test2
#if 0
launch_stream
<< "videotestsrc is-live=1 ! "
<< "video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! "
<< "nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! "
<< “omxh265enc ! fakesink sync=false async=false”;
endif
//mem leak test3
#if 0
launch_stream
<< "videotestsrc is-live=1 ! "
<< "video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! "
<< "nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! "
<< “fakesink sync=false async=false”;
endif

launch_string = launch_stream.str();

g_print("Using launch string: %s\n", launch_string.c_str());

GError *error = nullptr;
for(guint i=0; i<20000; i++) {
    printf("loop= %d\n", i);
    gst_pipeline  = (GstPipeline*) gst_parse_launch(launch_string.c_str(), &error);

    if (gst_pipeline == nullptr) {
        printf("Failed to parse launch: %s\n", error->message);
        return -1;
    }
    if(error) 
    {
        printf("Parse error[%d]\n",i);
        g_error_free(error);
    }

    printf("Set playing[%d]\n",i);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_PLAYING); 
    g_usleep(3*usec);
    printf("Send Eos[%d]\n",i);
    res=gst_element_send_event ((GstElement*)gst_pipeline, gst_event_new_eos ());
    if(!res)
    {
        printf("eos signal send failed\n");
    }
    else
    {
        printf("eos signal send success\n"); 
    }
    printf("get bus[%d]\n",i);
    bus = gst_pipeline_get_bus(GST_PIPELINE(gst_pipeline));
    printf("bus poll start[%d]\n",i);
    gst_bus_poll(bus, GST_MESSAGE_EOS, GST_CLOCK_TIME_NONE);
    printf("bus poll end &Set NULL[%d]\n",i);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);
    printf("Set unref[%d]\n",i);
    gst_object_unref(GST_OBJECT(gst_pipeline));
gst_object_unref (bus);
gst_pipeline=NULL;
bus=NULL;
    g_usleep(1*usec);
fflush(stdout);
}
g_main_loop_unref(main_loop);

g_print("going to exit \n");
return 0;

}
huilian@huilian-desktop:~/xxw/test1$ g++ -g test.cpp pkg-config --cflags --libs gstreamer-1.0
huilian@huilian-desktop:~/xxw/test1$

[DaneLLL]

Hi,
We apply this and run below script on r32.6.1/TX2 overnight. Don’t observe the issue:

#!/usr/bin/env python3

import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject, GLib

pipeline = None
bus = None
message = None

# initialize GStreamer
Gst.init(None)

for i in range(1, 65535):
    print("loop =",i," ")
    # build the pipeline
    pipeline = Gst.parse_launch(
        "videotestsrc is-live=1 ! video/x-raw,width=1280,height=960,format=YUY2,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=I420,width=1280,height=960,framerate=30/1 ! nvv4l2h265enc ! fakesink "
    )

    # start playing
    print("Switch to PLAYING state")
    pipeline.set_state(Gst.State.PLAYING)

    time.sleep(5)
    print("Send EoS")
    Gst.Element.send_event(pipeline, Gst.Event.new_eos())
    # wait until EOS or error
    bus = pipeline.get_bus()
    msg = bus.timed_pop_filtered(
        Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS)

    # free resources
    print("Switch to NULL state")
    pipeline.set_state(Gst.State.NULL)
    time.sleep(1)

The memory usage looks steady:
[start: PID 9960]

[13381 times]

Could you try it on r32.5.0? To check if the issue is specific to the version.

[happy886]

i test overnight,mem from 0.8% to 3.9% ,7795 times

[happy886]

i add GST_TRACE env, and run 50 times and end over with gst_deinit()
it printf leaks detected of GstCaps of encoder
0:03:33.169063483 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f9808a320, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);
0:03:33.169080602 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x55a16db230, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);
0:03:33.169099001 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x7f9405ea30, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);
0:03:33.169116377 30128 0x55a1686040 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0x55a1706c00, description=(string)video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, ref-count=(uint)1, trace=(string);

** (test5:30128): WARNING **: 12:25:00.173: Leaks detected

[happy886]

i add GST_TRACE env and over with gst_deinit,
nvv4l2h264enc,nvv4h265enc will appear leaks about GstCaps;but omxh265enc have not leaks !!!

[DaneLLL]

Hi,
Do you upgrade to r32.6.1 and do the test? Since we don’t see significant leak on r32.6.1, you may consider to upgrade to the version. We are deprecating omx plugins, but if the plugins work fine in your use-case, you may still consider to use it.

[happy886]

1. I did not upgrade to r32.6.1,just test on r32.5.0;

2. omxh265enc just for test,my project now use nvv4l2h265enc on r32.5.0,i do not want to update too often.

3.review the gst-v4l2 source,the gstv4l2h265enc.c use gst_static_caps_get(&src_template_caps) and not unref