# Deepstream GPU Memory Leak in nvv4l2decoder

**URL:** <https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852>\
**Category:** DeepStream SDK\
**Tags:** deepstream61, nvbugs\
**Created:** [July 14, 2022, 11:07pm UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852 "2022-07-14T23:07:04Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nimiwaribokoj](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nimiwaribokoj/32/67560_2.png) [@nimiwaribokoj](https://forums.developer.nvidia.com/u/nimiwaribokoj)\
**Post date:** [July 14, 2022, 11:07pm UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/1 "2022-07-14T23:07:04Z")

</div>

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU)**: dGPU, 3080 Ti  
**• DeepStream Version** : 6.1  
**• JetPack Version (valid for Jetson only)**: n/a  
**• TensorRT Version** : 8.4.0-1+cuda11.6  
**• NVIDIA GPU Driver Version (valid for GPU only)**: 515.48.07

I’m seeing an issue where, over time, the deepstream gstreamer pipeline begins to use all the memory available on the GPU. My pipeline (implemented using gstreamer-rs api) is equivalent to:

`uridecodebin | nvstreammux | queue | nvinfer | nvstreamdemux | queue | appsink`

For `nvstreammux` and `nvstreamdemux` I have set memory type to device. The stream is an rtsp source where in one case I’m using uridecodebin (and therefore rtspsrc) and in another case I’m using appsrc. The memory usage is not coming from rtspsrc as the memory leak is in the GPU memory.

When I check nvdia-smi, I can see the memory usage climb while the decoder utilization is only ~30%. cuda-memcheck doesn’t seem to be compatible with deepstream.

---

<div class="post-metadata">

**Author:** ![nimiwaribokoj](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nimiwaribokoj/32/67560_2.png) [@nimiwaribokoj](https://forums.developer.nvidia.com/u/nimiwaribokoj)\
**Post date:** [July 15, 2022, 2:07am UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/2 "2022-07-15T02:07:51Z")

</div>

Ok, we were able isolate the camera that is causing this issue. I can share the video via DM with an nvidia engineer who wishes to reproduce. Strangely this only occurs on Deepstream 6.1 with dGPU as our Deepstream 6.0 machine does not reproduce the issue.

A sample pipeline like so:

```auto
gst-launch-1.0 filesrc location=./2BH2PJqTApibLEGTv4Xjk8L70nn.mp4 ! qtdemux ! h264parse ! nvv4l2decoder cudadec-memtype=0 ! fakesink sync=false

```

Then if I analyze the output of `nvidia-smi`, the memory usage grows rapidly, nearly 4GiB to decode this video:

```auto
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 458818 C gst-launch-1.0 3757MiB |

```

Here is the ffprobe of the video:

```auto
ffprobe 2BH2PJqTApibLEGTv4Xjk8L70nn.mp4
ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil 56. 31.100 / 56. 31.100
  libavcodec 58. 54.100 / 58. 54.100
  libavformat 58. 29.100 / 58. 29.100
  libavdevice 58. 8.100 / 58. 8.100
  libavfilter 7. 57.100 / 7. 57.100
  libavresample 4. 0. 0 / 4. 0. 0
  libswscale 5. 5.100 / 5. 5.100
  libswresample 3. 5.100 / 3. 5.100
  libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2BH2PJqTApibLEGTv4Xjk8L70nn.mp4':
  Metadata:
    major_brand : isom
    minor_version : 512
    compatible_brands: isomiso2avc1mp41
  Duration: 00:30:00.00, start: 0.000000, bitrate: 3031 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 2688x1520 [SAR 1:1 DAR 168:95], 3031 kb/s, 10 fps, 10 tbr, 90k tbn, 20 tbc (default)

```

For reference here is a video that plays fine (notice this video has a higher resolution and bitrate):

```auto
$ ffprobe 2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4
ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil 56. 31.100 / 56. 31.100
  libavcodec 58. 54.100 / 58. 54.100
  libavformat 58. 29.100 / 58. 29.100
  libavdevice 58. 8.100 / 58. 8.100
  libavfilter 7. 57.100 / 7. 57.100
  libavresample 4. 0. 0 / 4. 0. 0
  libswscale 5. 5.100 / 5. 5.100
  libswresample 3. 5.100 / 3. 5.100
  libpostproc 55. 5.100 / 55. 5.100
[h264 @ 0x55c8c9fcdbc0] Missing reference picture, default is 0
[h264 @ 0x55c8c9fcdbc0] decode_slice_header error
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4':
  Metadata:
    major_brand : isom
    minor_version : 512
    compatible_brands: isomiso2avc1mp41
  Duration: 00:29:59.97, start: 0.000000, bitrate: 3624 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 3648x2052 [SAR 1:1 DAR 16:9], 3624 kb/s, 15.21 fps, 15.25 tbr, 90k tbn, 30 tbc (default)

```

then gst-launch:

```auto
gst-launch-1.0 filesrc location=./2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4 ! qtdemux ! h264parse ! nvv4l2decoder cudadec-memtype=0 ! fakesink sync=false

```

Memory is never above 300MB

```auto
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 458908 C gst-launch-1.0 309MiB |

```

**Edit** : We were able to see that NVDEC produces invalid frames for this stream as well; so it might not be related to specifically deepstream. The deepstream-6.0 machine is at driver version `510.47.03 + cuda 11.4` while the problem machines are at `515.48.07 + cuda 11.6`

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [July 15, 2022, 8:18am UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/4 "2022-07-15T08:18:42Z")

</div>

Please share the video file to us for analysis.

---

<div class="post-metadata">

**Author:** ![nimiwaribokoj](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nimiwaribokoj/32/67560_2.png) [@nimiwaribokoj](https://forums.developer.nvidia.com/u/nimiwaribokoj)\
**Post date:** [July 15, 2022, 9:08am UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/5 "2022-07-15T09:08:28Z")

</div>

I have shared the video via the message feature. Please let me know if you are able to reproduce the issue. For more context:

Machine 1:

```auto
Ubuntu 20.04.4 LTS
Linux 5.4.0-121-generic 
NVIDIA-SMI 515.48.07
CUDA 11.6
Deepstream 6.1
NVIDIA GeForce RTX 3080 Ti

```

Machine 2:

```auto
Ubuntu 20.04.2 LTS
Linux 5.13.0-51-generic
NVIDIA-SMI 515.48.07
CUDA 11.6
Deepstream 6.1
NVIDIA GeForce RTX 3080

```

---

<div class="post-metadata">

**Author:** ![nimiwaribokoj](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nimiwaribokoj/32/67560_2.png) [@nimiwaribokoj](https://forums.developer.nvidia.com/u/nimiwaribokoj)\
**Post date:** [July 19, 2022, 9:26pm UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/6 "2022-07-19T21:26:47Z")

</div>

Any update or acknowledgement of the issue?

---

<div class="post-metadata">

**Author:** ![nimiwaribokoj](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nimiwaribokoj/32/67560_2.png) [@nimiwaribokoj](https://forums.developer.nvidia.com/u/nimiwaribokoj)\
**Post date:** [July 26, 2022, 10:53pm UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/8 "2022-07-26T22:53:47Z")

</div>

Just replying again to see if this issue has been looked into.

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [August 1, 2022, 10:57am UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/10 "2022-08-01T10:57:01Z")

</div>

I can reproduce the issue. We will look into it.

---

<div class="post-metadata">

**Author:** ![yingliu](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yingliu/32/136703_2.png) [@yingliu](https://forums.developer.nvidia.com/u/yingliu)\
**Post date:** [August 23, 2022, 5:52am UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/11 "2022-08-23T05:52:18Z")

</div>

Issue will be fixed in next release.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [September 6, 2022, 5:53am UTC](https://forums.developer.nvidia.com/t/deepstream-gpu-memory-leak-in-nvv4l2decoder/220852/12 "2022-09-06T05:53:00Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.