Nvv4l2h265enc memory leak when changing caps

[JimenaSalas]

Hi everyone,

I’m using Jetpack 4.6 and I was working on some tests for an application when I ran into a memory leak issue which seems to be related to the encoder. This happens when caps (framerate, resolution, etc) are changed while in playing state. Here’s an example to reproduce it:

#!/usr/bin/env python3

import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject, GLib

pipeline = None

# initialize GStreamer
Gst.init(None)
pipeline = Gst.parse_launch(
        "videotestsrc is-live=1 ! video/x-raw,width=1280,height=720 ! capsfilter name=framerate caps=\"video/x-raw,framerate=60/1\" ! nvvidconv ! nvv4l2h265enc ! fakesink "
    )

capsfilter = pipeline.get_by_name("framerate")

for i in range(1, 1000):
    print("loop =",i," ")

    # start playing
    pipeline.set_state(Gst.State.PLAYING)
    pipeline.get_state(Gst.CLOCK_TIME_NONE)


    capsfilter.set_property("caps", Gst.Caps.from_string("video/x-raw, framerate=60/1"))
    time.sleep(1)
    capsfilter.set_property("caps", Gst.Caps.from_string("video/x-raw, framerate=30/1"))
    time.sleep(1)

    pipeline.set_state(Gst.State.NULL)
    pipeline.get_state(Gst.CLOCK_TIME_NONE)

In every iteration I see an increase of approximately 60MB of memory. I tried to run valgrind but it seems that the issue points to some private code:
log_valgrind.txt (7.3 KB)

Is there any known solution? (We would like to avoid stopping the pipeline to set the caps)

Thanks in advance.

[DaneLLL]

Hi,
Looks like changing caps will trigger caps negotiation. It is not expected if the pipeline is not terminated and re-initialized We have the code about changing frame rate in 01_video_encode and a possible solution is to add one more property to nvv4l2h264enc/nvv4l2h265enc. So that we can set it once the source changes frame rate. We will check this and update.

[JimenaSalas]

Hi @DaneLLL ,

Thanks! It would be great if you could take a look since it happens also with resolution changes and basically anything else that triggers caps renegotiation in the pipeline.

[DaneLLL]

Hi,
Changing bitrate/framerate in runtime should work without re-initializing the pipeline. But for changing resolution, would need to re-initialize the pipeline.

[JimenaSalas]

Both of them, resolution and framerate changes work in runtime (the results are visible when displaying), the only problem is that memory leak that is left behind.

[DaneLLL]

Hi,
For changing frame rate, please apply the patch and rebuild/replace libgstnvvideo4linux2.so:

diff --git a/gst-v4l2/gstv4l2videoenc.c b/gst-v4l2/gstv4l2videoenc.c
index 1120f93..54b5362 100644
--- a/gst-v4l2/gstv4l2videoenc.c
+++ b/gst-v4l2/gstv4l2videoenc.c
@@ -128,7 +128,8 @@ enum
   PROP_MAX_PERF,
   PROP_IDR_FRAME_INTERVAL,
   PROP_FORCE_INTRA,
-  PROP_FORCE_IDR
+  PROP_FORCE_IDR,
+  PROP_FRATE
 #endif
 };
 
@@ -235,6 +236,20 @@ gst_v4l2_video_enc_set_property_tegra (GObject * object,
       self->peak_bitrate = g_value_get_uint (value);
       break;
 
+    case PROP_FRATE:
+    {
+      struct v4l2_streamparm streamparm;
+      memset (&streamparm, 0x00, sizeof (struct v4l2_streamparm));
+      streamparm.type = self->v4l2output->type;
+      /* Note: V4L2 wants the frame interval, we have the frame rate */
+      streamparm.parm.capture.timeperframe.denominator =
+          gst_value_get_fraction_numerator (value);
+      streamparm.parm.capture.timeperframe.numerator =
+          gst_value_get_fraction_denominator (value);
+      self->v4l2output->ioctl (self->v4l2output->video_fd, VIDIOC_S_PARM, &streamparm);
+      break;
+    }
+
     case PROP_QUANT_RANGE:
       gst_v4l2_video_enc_parse_quantization_range (self,
           g_value_get_string (value));
@@ -418,6 +433,9 @@ gst_v4l2_video_enc_get_property_tegra (GObject * object,
     case PROP_IDR_FRAME_INTERVAL:
       g_value_set_uint (value, self->idrinterval);
       break;
+
+    case PROP_FRATE:
+      break;
 #endif
 
       /* By default read from output */
@@ -1698,6 +1716,14 @@ gst_v4l2_video_enc_class_init (GstV4l2VideoEncClass * klass)
           G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS |
           GST_PARAM_MUTABLE_READY));
 
+  g_object_class_install_property (gobject_class, PROP_FRATE,
+      gst_param_spec_fraction ("framerate", "On-the-fly frame rate",
+          "Set frame rate on the fly",
+          0, 1, G_MAXINT, 1,
+          30, 1,
+          G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS | GST_PARAM_MUTABLE_PLAYING)
+      );
+
   g_object_class_install_property (gobject_class, PROP_INTRA_FRAME_INTERVAL,
       g_param_spec_uint ("iframeinterval", "Intra Frame interval",
           "Encoding Intra Frame occurance frequency",

And please also apply this:
Memory Leak (Alloc/free mismatch) in Tegra multimedia API (encoder) - #6 by DaneLLL

We don’t support changing resolution in runtime by default. It is not handled and verified. For changing resolution, we would suggest re-initialize the whole pipeline. If you have to enable this use-case, may need to check the source code and see if it is possible to do customization.

[JimenaSalas]

Hi @DaneLLL ,

Thanks for the patch but I don’t think it helps in this case since it doesn’t handle any renegotiation, so other elements of the pipeline will not know that the framerate changed. For example if you run this pipeline:

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1280,height=720 ! nvvidconv ! perf ! nvv4l2h265enc name=enc framerate=60/1 ! perf ! fakesink -v

The negotiated framerate reports 30, which is the default for videotestsrc, even though I set 60 on the encoder property. This is the output of the negotiated caps:

/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive

And here’s an output using gst-perf. It also reports ~30.

INFO:
perf: perf0; timestamp: 0:41:17.929062529; bps: 241920,000; mean_bps: 0,000; fps: 29,929; mean_fps: 29,929
INFO:
perf: perf1; timestamp: 0:41:17.945732618; bps: 5505608,000; mean_bps: 0,000; fps: 29,684; mean_fps: 29,684

I tried your suggestion for renaming the argus library but it didn’t make any difference for the leaks.

[DaneLLL]

Hi,
The videotestsrc is for testing. For real source such as nvarguscamerasrc, you can add code to change frame rate of source and set the property to encoder to get desired bitrate.