Gstreamer nv3dsink memory gstcaps leaks

pascal.segret · February 16, 2024, 1:29pm

Board: Jetson Orin Nx
JetPack 6.0

I’m working on a project with a Jetson Orin board, and I’m having a memory leak problem,
I can reproduce the problem with the command line:

jetson@JetsonNX16:~/gstreamer_flux$ GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks" gst-launch-1.0 videotestsrc  ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nv3dsink
0:00:00.023006296 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab054519c0 (latency)
0:00:00.023074650 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab05451a80 (log)
0:00:00.023092443 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab05451b40 (rusage)
0:00:00.023106683 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab05451c00 (stats)
0:00:00.023118715 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab05451cc0 (leaks)
0:00:00.023130012 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab05451d80 (factories)
0:00:00.023204478 599189 0xaaab05609d60 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;", description=(structure)"value\,\ type\=\(type\)gchararray\;", ref-count=(structure)"value\,\ type\=\(type\)guint\;", trace=(structure)"value\,\ type\=\(type\)gchararray\;";
0:00:00.023228671 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.023255808 599189 0xaaab05609d60 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-refings.class, ts=(structure)"value\,\ type\=\(type\)guint64\;", type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;", description=(structure)"value\,\ type\=\(type\)gchararray\;", ref-count=(structure)"value\,\ type\=\(type\)guint\;", trace=(structure)"value\,\ type\=\(type\)gchararray\;";
0:00:00.023270272 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-refings, ts=(guint64)%lu, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.023287041 599189 0xaaab05609d60 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-added.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;";
0:00:00.023297025 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-added, type-name=(string)%s, address=(gpointer)%p;
0:00:00.023318113 599189 0xaaab05609d60 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-removed.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;";
0:00:00.023329762 599189 0xaaab05609d60 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-removed, type-name=(string)%s, address=(gpointer)%p;
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:02.164977411
Setting pipeline to NULL ...
Freeing pipeline ...
0:00:02.415787914 599189 0xaaab05609d60 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff8c002450, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], multiview-mode=(string){ mono, left, right }, ref-count=(uint)1, trace=(string);
0:00:02.416309978 599189 0xaaab05609d60 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff8c0025e0, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)[ 1/2147483647, 2147483647/1 ], interlace-mode=(string)progressive, ref-count=(uint)1, trace=(string);

** (gst-launch-1.0:599189): WARNING **: 14:20:00.971: Leaks detected and logged under GST_DEBUG=GST_TRACER:7
jetson@JetsonNX16:~/gstreamer_flux$

If I validate the traces “GST_DEBUG=“8,GST_TRACER:7” GST_TRACERS=“leaks””
I see a memory allocation by the “nv3dsink” plugin which is never freed.

I don’t have the problem if I use the “autovideosink” plugin

DaneLLL · February 19, 2024, 2:50am

Hi,
Jetpack 6.0 DP is developer preview and there may be some issues/bugs. Could you try Jetpack 5.1.2 and check if you observe the same?

pascal.segret · February 19, 2024, 7:50am

Hi,

In Jetpack Version 5.1.2, it’s the same error.

jetson@jetson-177:~$ sudo apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 5.1.2-b104
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 5.1.2-b104), nvidia-jetpack-dev (= 5.1.2-b104)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_5.1.2-b104_arm64.deb
Size: 29304
SHA256: fda2eed24747319ccd9fee9a8548c0e5dd52812363877ebe90e223b5a6e7e827
SHA1: 78c7d9e02490f96f8fbd5a091c8bef280b03ae84
MD5sum: 6be522b5542ab2af5dcf62837b34a5f0
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

jetson@jetson-177:~$ GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks" gst-launch-1.0 videotestsrc  ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nv3dsink
0:00:00.015250851  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf5ca79c0 (latency)
0:00:00.015338437  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf5ca7a80 (log)
0:00:00.015361094  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf5ca7b40 (rusage)
0:00:00.015379782  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf5ca7c00 (stats)
0:00:00.015397191  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf5ca7cc0 (leaks)
0:00:00.015524681  3720 0xaaaaf5e25400 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(type\)gpointer\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(type\)guint\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.015558698  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.015605323  3720 0xaaaaf5e25400 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: object-refings.class, ts=(structure)"value\,\ type\=\(type\)guint64\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", type-name=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(type\)gpointer\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(type\)guint\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.015623915  3720 0xaaaaf5e25400 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: object-refings, ts=(guint64)%lu, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:05.955982643
Setting pipeline to NULL ...
Freeing pipeline ...
0:00:06.103500799  3720 0xaaaaf5e25400 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff70004050, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], multiview-mode=(string){ mono, left, right }, ref-count=(uint)1, trace=(string);
0:00:06.103617729  3720 0xaaaaf5e25400 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff700041e0, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)[ 1/2147483647, 2147483647/1 ], interlace-mode=(string)progressive, ref-count=(uint)1, trace=(string);

** (gst-launch-1.0:3720): WARNING **: 08:47:32.533: Leaks detected
jetson@jetson-177:~$

DaneLLL · February 21, 2024, 4:52am

Hi,
Thanks for reporting it. We will check further.

pascal.segret · February 21, 2024, 10:56am

Hi,

Good news, but in my application, I have video streams which change periodically and they are sent to nv3dsink for display which causes significant memory leaks and my application is not viable.
I couldn’t find the nv3dsink sources in plubic_sources to search for the problems?

DaneLLL · February 22, 2024, 1:05am

Hi,
The source code of nv3dsink is public. Please download the package

https://developer.nvidia.com/embedded/jetson-linux-r3541
Driver Package (BSP) Sources

and check libgstnvvideosinks_src.tbz2

pascal.segret · February 23, 2024, 10:17am

Hi,

I looked at the nv3dsink source code and after some tracing I noticed that the problem comes from the gst_nv3dsink_set_caps() routine in nv3dsink.c, it uses the gst_mini_object_replace() function which increments the reference count of the caps.
We can fix the problem in nv3dsink with

  g_mutex_unlock (&nv3dsink->win_handle_lock);

  gboolean bUnref = (nv3dsink->configured_caps != caps);
  gst_caps_replace (&nv3dsink->configured_caps, caps);
  if (caps && bUnref)  gst_caps_unref (caps);

  return TRUE;
}

Looking at the code of the gst_mini_object_replace() function, we can clearly see, the reference count of @olddata is decreased and the reference count of @newdata is increased.

/**
 * gst_mini_object_replace:
 * @olddata: (inout) (transfer full) (nullable): pointer to a pointer to a
 *     mini-object to be replaced
 * @newdata: (allow-none): pointer to new mini-object
 *
 * Atomically modifies a pointer to point to a new mini-object.
 * The reference count of @olddata is decreased and the reference count of
 * @newdata is increased.
 *
 * Either @newdata and the value pointed to by @olddata may be %NULL.
 *
 * Returns: %TRUE if @newdata was different from @olddata
 */
gboolean
gst_mini_object_replace (GstMiniObject ** olddata, GstMiniObject * newdata)
{
  GstMiniObject *olddata_val;

  g_return_val_if_fail (olddata != NULL, FALSE);

  GST_CAT_TRACE (GST_CAT_REFCOUNTING, "replace %p (%d) with %p (%d)",
      *olddata, *olddata ? (*olddata)->refcount : 0,
      newdata, newdata ? newdata->refcount : 0);

  olddata_val = g_atomic_pointer_get ((gpointer *) olddata);

  if (G_UNLIKELY (olddata_val == newdata))
    return FALSE;

  if (newdata)
    gst_mini_object_ref (newdata);

  while (G_UNLIKELY (!g_atomic_pointer_compare_and_exchange ((gpointer *)
              olddata, olddata_val, newdata))) {
    olddata_val = g_atomic_pointer_get ((gpointer *) olddata);
    if (G_UNLIKELY (olddata_val == newdata))
      break;
  }

  if (olddata_val)
    gst_mini_object_unref (olddata_val);

  return olddata_val != newdata;
}

But, I looked deeper into the traces, we see some bad uses of gst_mini_object_replace()…

The problem happens when the new object and the old object are different and not null, this increment is not managed in certain plugins like nv3dsink

What causes memory leaks… despairing

DaveYYY · February 26, 2024, 5:23am

Hi,

Do you have any further updates/finding on this issue?
The patch does look reasonable, but we still got this after applying the patch on both 5.1.2 and 6.0:

0:00:00.082476974  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab041d21c0 (latency)
0:00:00.082559562  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab041d2280 (log)
0:00:00.082584873  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab041d2340 (rusage)
0:00:00.082604937  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab041d2400 (stats)
0:00:00.082622664  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab041d24c0 (leaks)
0:00:00.082639015  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracer.c:159:gst_tracer_register:<registry0> update existing feature 0xaaab041d2580 (factories)
0:00:00.082745762  5044 0xaaab042f6870 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;", description=(structure)"value\,\ type\=\(type\)gchararray\;", ref-count=(structure)"value\,\ type\=\(type\)guint\;", trace=(structure)"value\,\ type\=\(type\)gchararray\;";
0:00:00.082785505  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.082824927  5044 0xaaab042f6870 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-refings.class, ts=(structure)"value\,\ type\=\(type\)guint64\;", type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;", description=(structure)"value\,\ type\=\(type\)gchararray\;", ref-count=(structure)"value\,\ type\=\(type\)guint\;", trace=(structure)"value\,\ type\=\(type\)gchararray\;";
0:00:00.082846302  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-refings, ts=(guint64)%lu, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.082869213  5044 0xaaab042f6870 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-added.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;";
0:00:00.082882780  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-added, type-name=(string)%s, address=(gpointer)%p;
0:00:00.082903100  5044 0xaaab042f6870 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-removed.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;";
0:00:00.082916219  5044 0xaaab042f6870 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-removed, type-name=(string)%s, address=(gpointer)%p;
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:01.556713011
Setting pipeline to NULL ...

(gst-launch-1.0:5044): GStreamer-CRITICAL **: 05:20:34.710: gst_mini_object_unref: assertion 'GST_MINI_OBJECT_REFCOUNT_VALUE (mini_object) > 0' failed
Freeing pipeline ...
0:00:01.871717394  5044 0xaaab042f6870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff580025e0, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)[ 1/2147483647, 2147483647/1 ], interlace-mode=(string)progressive, ref-count=(uint)1, trace=(string);
0:00:01.871810542  5044 0xaaab042f6870 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff58002450, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], multiview-mode=(string){ mono, left, right }, ref-count=(uint)1, trace=(string);

** (gst-launch-1.0:5044): WARNING **: 05:20:34.799: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Is it working in your environment?

pascal.segret · February 26, 2024, 7:24am

Hi,
At first, I thought the modification worked, but looking deeper into the gst_caps_replace() function, there is still the problem,
I’m still working on it…

DaveYYY · February 26, 2024, 7:37am

Sure.
I think the problem lies somewhere else, because the assertion failure means the additional unref call is not needed.
I will also look into the code more.

pascal.segret · February 26, 2024, 9:33am

yes I think like you

DaveYYY · February 27, 2024, 6:17am

Hi,

Some findings on the issue:
One of the memory leaks actually happens in nvvidconv.

If you remove nvvidconv from the pipeline (removing (memory:NVMM) is also needed as no transform is done to support NVMM memory now), then the number of reported memory leaks is down to one.

DISPLAY=:0 GST_DEBUG=“GST_TRACER:7” GST_TRACERS=“leaks” gst-launch-1.0 videotestsrc ! ‘video/x-raw, format=RGBA’ ! nv3dsink

0:00:00.014010735  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaae73881c0 (latency)
0:00:00.014089809  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaae7388280 (log)
0:00:00.014109938  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaae7388340 (rusage)
0:00:00.014125810  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaae7388400 (stats)
0:00:00.014139443  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaae73884c0 (leaks)
0:00:00.014237942  5522 0xaaaae7505580 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(type\)gpointer\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(type\)guint\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.014268982  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.014306007  5522 0xaaaae7505580 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: object-refings.class, ts=(structure)"value\,\ type\=\(type\)guint64\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", type-name=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(type\)gpointer\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(type\)guint\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.014323192  5522 0xaaaae7505580 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: object-refings, ts=(guint64)%lu, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:01.546383818
Setting pipeline to NULL ...
Freeing pipeline ...
0:00:01.748659399  5522 0xaaaae7505580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xaaaae74ce6d0, description=(string)video/x-raw, format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], multiview-mode=(string){ mono, left, right }, ref-count=(uint)1, trace=(string);

** (gst-launch-1.0:5522): WARNING **: 06:14:27.694: Leaks detected

nvvidconv is also open-source, and the source code is available in gst-nvvidconv_src.tbz2.
You can also take a look.

pascal.segret · February 27, 2024, 6:49am

Hi,
I loaded the nvvidconv sources yesterday, I noticed that the GstCaps used by nv3dsink were related to nvvidconv.
I’m looking at the nvvidconv sources. :-)

pascal.segret · February 27, 2024, 9:18am

Hi,

With the test

GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks" gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nv3dsink

When we stop the test with a CTL-C, there is no call in gstnvconv to the function gst_nv_filter_buffer_pool_stop() to free the “pool->caps” and we have the memory leak, we can see it by adding

5,GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks

to see the traces.

So, I went on another test, with playing a video that stops at the end of the film:

GST_DEBUG="5,GST_TRACER:7" GST_TRACERS="leaks"
gst-launch-1.0 filesrc location= /usr/src/jetson_multimedia_api/data/Video/sample_outdoor_car_1080p_10fps.h264 ! h264parse! tail ! nvv4l2decoder! nvvidconv! 'video/x-raw(memory:NVMM), format=RGBA'! tail ! nv3dsink

In this case, we have gst_nv_filter_buffer_pool_stop() of gstnvconv,
but we still have memory leak “GstCaps”,

I will continue with this test to search for memory leaks.

DaveYYY · February 29, 2024, 2:55am

Hi,

I do see gst_nv_filter_buffer_pool_stop() getting called in both the case of EOS and CTRL-C.
Do you have the log as text files?

Also, while playing a real video file stream, it seems we are getting much more memory leaks:

DISPLAY=:0 GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks" gst-launch-1.0 filesrc location= /usr/src/jetson_multimedia_api/data/Video/sample_outdoor_car_1080p_10fps.h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), format=RGBA'! nv3dsink

(I dont’t really know what tail is in your pipeline.)

nvidia@tegra-ubuntu:~/Downloads$ DISPLAY=:0 GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks" gst-launch-1.0 filesrc location= /usr/src/jetson_multimedia_api/data/Video/sample_outdoor_car_1080p_10fps.h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), format=RGBA'! nv3dsink
0:00:00.014127054 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaab053fb4c0 (latency)
0:00:00.014201551 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaab053fb400 (log)
0:00:00.014216783 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaab053fb340 (rusage)
0:00:00.014228335 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaab053fb280 (stats)
0:00:00.014239375 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaab053fb1c0 (leaks)
0:00:00.014342991 11703 0xaaab05577580 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(type\)gpointer\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(type\)guint\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.014377839 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:00.014411503 11703 0xaaab05577580 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: object-refings.class, ts=(structure)"value\,\ type\=\(type\)guint64\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", type-name=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(type\)gpointer\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(type\)guint\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.014425679 11703 0xaaab05577580 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: object-refings, ts=(guint64)%lu, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:59.100760583
Setting pipeline to NULL ...
Freeing pipeline ...
0:00:59.513012328 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xaaab056d7e50, description=(string)0xaaab056d7e50, ref-count=(uint)0, trace=(string);
0:00:59.513088968 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009aec0, description=(string)0xffff8009aec0, ref-count=(uint)0, trace=(string);
0:00:59.513108296 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009a890, description=(string)0xffff8009a890, ref-count=(uint)0, trace=(string);
0:00:59.513122760 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009a9b0, description=(string)0xffff8009a9b0, ref-count=(uint)0, trace=(string);
0:00:59.513138696 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009aad0, description=(string)0xffff8009aad0, ref-count=(uint)11, trace=(string);
0:00:59.513153192 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff800a40d0, description=(string)0xffff800a40d0, ref-count=(uint)0, trace=(string);
0:00:59.513166536 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009abf0, description=(string)0xffff8009abf0, ref-count=(uint)4072713477, trace=(string);
0:00:59.513183784 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff800a41f0, description=(string)0xffff800a41f0, ref-count=(uint)0, trace=(string);
0:00:59.513201544 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009ad10, description=(string)0xffff8009ad10, ref-count=(uint)0, trace=(string);
0:00:59.513215112 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009ae30, description=(string)0xffff8009ae30, ref-count=(uint)0, trace=(string);
0:00:59.513227656 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xaaab056d7ee0, description=(string)0xaaab056d7ee0, ref-count=(uint)0, trace=(string);
0:00:59.513241128 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009a800, description=(string)0xffff8009a800, ref-count=(uint)0, trace=(string);
0:00:59.513253320 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009a920, description=(string)0xffff8009a920, ref-count=(uint)0, trace=(string);
0:00:59.513266056 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009aa40, description=(string)0xffff8009aa40, ref-count=(uint)0, trace=(string);
0:00:59.513279977 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff800a4040, description=(string)0xffff800a4040, ref-count=(uint)0, trace=(string);
0:00:59.513292841 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009ab60, description=(string)0xffff8009ab60, ref-count=(uint)2148156608, trace=(string);
0:00:59.513306537 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff800a4160, description=(string)0xffff800a4160, ref-count=(uint)0, trace=(string);
0:00:59.513319273 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009ac80, description=(string)0xffff8009ac80, ref-count=(uint)2148156320, trace=(string);
0:00:59.513332617 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)(null), address=(gpointer)0xffff8009ada0, description=(string)0xffff8009ada0, ref-count=(uint)0, trace=(string);
0:00:59.513348361 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff800041e0, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], ref-count=(uint)1, trace=(string);
0:00:59.513365353 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff80004230, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)10/1, pixel-aspect-ratio=(fraction)[ 1/2147483647, 2147483647/1 ], ref-count=(uint)1, trace=(string);
0:00:59.513380617 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff80003630, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], ref-count=(uint)1, trace=(string);
0:00:59.513396393 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff80004050, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], ref-count=(uint)1, trace=(string);
0:00:59.515178639 11703 0xaaab05577580 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)0xffff80003450, description=(string)video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)[ 1/2147483647, 2147483647/1 ], framerate=(fraction)10/1, nvbuf-memory-type=(string)nvbuf-mem-surface-array, gpu-id=(int)0, ref-count=(uint)1, trace=(string);

** (gst-launch-1.0:11703): WARNING **: 02:45:30.028: Leaks detected

Is it also what you are getting?
I think we may first focus on the case of videotestsrc with CTRL-C.

pascal.segret · February 29, 2024, 7:38am

HI

Yes, I have the log as text files.

when I add the DEBUG in GST_DEBUG, I did not see the trace “GST_DEBUG_OBJECT (pool, “stop”);” on stopping with a CTRL-C.

With the film, yes, there are many more errors…

I think there is a problem in nvvidconv and the negotiation of GstCaps.

DaveYYY · March 5, 2024, 3:09am

Hi,

It’s indeed not related to nvvidconv, and only happens inside nv3dsink.
Please apply the patch and try again.
The memory leak is gone per our local test.

diff --git a/gst-plugins-nv-video-sinks/nv3dsink/gstnv3dsink.c b/gst-plugins-nv-video-sinks/nv3dsink/gstnv3dsink.c
index fc6a440..b03f70b 100644
--- a/gst-plugins-nv-video-sinks/nv3dsink/gstnv3dsink.c
+++ b/gst-plugins-nv-video-sinks/nv3dsink/gstnv3dsink.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2018-2022, NVIDIA CORPORATION.  All rights reserved.
+ * Copyright (c) 2018-2024, NVIDIA CORPORATION.  All rights reserved.
  *
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public
@@ -333,8 +333,10 @@ gst_nv3dsink_get_caps (GstBaseSink * bsink, GstCaps * filter)
 
   caps = gst_nv_video_context_get_caps (nv3dsink->context);
   if (caps) {
+    tmp = result;
     result = gst_caps_intersect (result, caps);
     gst_caps_unref (caps);
+    gst_caps_unref (tmp);
   }
 
   GST_DEBUG_OBJECT (bsink, "returning caps: %" GST_PTR_FORMAT, result);

pascal.segret · March 5, 2024, 7:07am

Hi

I tried the patch and indeed, there are no more memory leaks in nv3dsink

Great!

system · March 27, 2024, 7:05am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Possible memory leak in nvvidconv gstreamer plugin or decoder? Jetson Nano gstreamer	4	1810	October 18, 2021
Memory leak in nveglglessink - Deepstream 5.1 DeepStream SDK gstreamer , jetson , deepstream	2	827	November 30, 2021
Gst_element_set_state() is hanging while setting the element to NULL DeepStream SDK	15	1303	August 17, 2023
Gstreamer Pipeline issue : PosixMemMap:84 mmap failed Jetson Orin Nano gstreamer	15	667	April 25, 2024
Nvv4l2h265enc crash problem Jetson TX2 gstreamer	23	2866	February 23, 2022
Restarting pipeline on DeepStream 5.0 DeepStream SDK	15	2590	October 12, 2021
[DEEPSTREAM] nvds_obj_enc_process Segmentation Fault on Yolo and Jetson nano DeepStream SDK camera , gstreamer , encoder , jetson , deepstream	11	53	October 9, 2024
Rtsp source issues with some cameras DeepStream SDK camera , gstreamer	10	1304	September 6, 2022
Deadlock in gst_element_release_request_pad with nvstreammux DeepStream SDK	29	3524	October 12, 2021
Memory leaks detected in GStreamer elements (omxh264 or nvvidconv) Jetson TX2	14	5029	October 18, 2021

Gstreamer nv3dsink memory gstcaps leaks

Related topics