Deepstream GPU Memory Leak in nvv4l2decoder

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): dGPU, 3080 Ti
• DeepStream Version: 6.1
• JetPack Version (valid for Jetson only): n/a
• TensorRT Version: 8.4.0-1+cuda11.6
• NVIDIA GPU Driver Version (valid for GPU only): 515.48.07

I’m seeing an issue where, over time, the deepstream gstreamer pipeline begins to use all the memory available on the GPU. My pipeline (implemented using gstreamer-rs api) is equivalent to:

uridecodebin | nvstreammux | queue | nvinfer | nvstreamdemux | queue | appsink

For nvstreammux and nvstreamdemux I have set memory type to device. The stream is an rtsp source where in one case I’m using uridecodebin (and therefore rtspsrc) and in another case I’m using appsrc. The memory usage is not coming from rtspsrc as the memory leak is in the GPU memory.

When I check nvdia-smi, I can see the memory usage climb while the decoder utilization is only ~30%. cuda-memcheck doesn’t seem to be compatible with deepstream.

Ok, we were able isolate the camera that is causing this issue. I can share the video via DM with an nvidia engineer who wishes to reproduce. Strangely this only occurs on Deepstream 6.1 with dGPU as our Deepstream 6.0 machine does not reproduce the issue.

A sample pipeline like so:

gst-launch-1.0 filesrc location=./2BH2PJqTApibLEGTv4Xjk8L70nn.mp4  ! qtdemux ! h264parse ! nvv4l2decoder cudadec-memtype=0 ! fakesink sync=false

Then if I analyze the output of nvidia-smi, the memory usage grows rapidly, nearly 4GiB to decode this video:

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    458818      C   gst-launch-1.0                   3757MiB |

Here is the ffprobe of the video:

ffprobe 2BH2PJqTApibLEGTv4Xjk8L70nn.mp4
ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2BH2PJqTApibLEGTv4Xjk8L70nn.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
  Duration: 00:30:00.00, start: 0.000000, bitrate: 3031 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 2688x1520 [SAR 1:1 DAR 168:95], 3031 kb/s, 10 fps, 10 tbr, 90k tbn, 20 tbc (default)

For reference here is a video that plays fine (notice this video has a higher resolution and bitrate):

$ ffprobe 2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4
ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[h264 @ 0x55c8c9fcdbc0] Missing reference picture, default is 0
[h264 @ 0x55c8c9fcdbc0] decode_slice_header error
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
  Duration: 00:29:59.97, start: 0.000000, bitrate: 3624 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 3648x2052 [SAR 1:1 DAR 16:9], 3624 kb/s, 15.21 fps, 15.25 tbr, 90k tbn, 30 tbc (default)

then gst-launch:

gst-launch-1.0 filesrc location=./2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4  ! qtdemux ! h264parse ! nvv4l2decoder cudadec-memtype=0 ! fakesink sync=false

Memory is never above 300MB

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    458908      C   gst-launch-1.0                    309MiB |

Edit: We were able to see that NVDEC produces invalid frames for this stream as well; so it might not be related to specifically deepstream. The deepstream-6.0 machine is at driver version 510.47.03 + cuda 11.4 while the problem machines are at 515.48.07 + cuda 11.6

Please share the video file to us for analysis.

I have shared the video via the message feature. Please let me know if you are able to reproduce the issue. For more context:

Machine 1:

Ubuntu 20.04.4 LTS
Linux 5.4.0-121-generic 
NVIDIA-SMI 515.48.07
CUDA 11.6
Deepstream 6.1
NVIDIA GeForce RTX 3080 Ti

Machine 2:

Ubuntu 20.04.2 LTS
Linux 5.13.0-51-generic
NVIDIA-SMI 515.48.07
CUDA 11.6
Deepstream 6.1
NVIDIA GeForce RTX 3080

Any update or acknowledgement of the issue?

Just replying again to see if this issue has been looked into.

I can reproduce the issue. We will look into it.