Ok, we were able isolate the camera that is causing this issue. I can share the video via DM with an nvidia engineer who wishes to reproduce. Strangely this only occurs on Deepstream 6.1 with dGPU as our Deepstream 6.0 machine does not reproduce the issue.
A sample pipeline like so:
gst-launch-1.0 filesrc location=./2BH2PJqTApibLEGTv4Xjk8L70nn.mp4 ! qtdemux ! h264parse ! nvv4l2decoder cudadec-memtype=0 ! fakesink sync=false
Then if I analyze the output of nvidia-smi, the memory usage grows rapidly, nearly 4GiB to decode this video:
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 458818 C gst-launch-1.0 3757MiB |
Here is the ffprobe of the video:
ffprobe 2BH2PJqTApibLEGTv4Xjk8L70nn.mp4
ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2BH2PJqTApibLEGTv4Xjk8L70nn.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
Duration: 00:30:00.00, start: 0.000000, bitrate: 3031 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 2688x1520 [SAR 1:1 DAR 168:95], 3031 kb/s, 10 fps, 10 tbr, 90k tbn, 20 tbc (default)
For reference here is a video that plays fine (notice this video has a higher resolution and bitrate):
$ ffprobe 2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4
ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[h264 @ 0x55c8c9fcdbc0] Missing reference picture, default is 0
[h264 @ 0x55c8c9fcdbc0] decode_slice_header error
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
Duration: 00:29:59.97, start: 0.000000, bitrate: 3624 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 3648x2052 [SAR 1:1 DAR 16:9], 3624 kb/s, 15.21 fps, 15.25 tbr, 90k tbn, 30 tbc (default)
then gst-launch:
gst-launch-1.0 filesrc location=./2BJkOiL9kBYVdLa1IIDBmYUsgCO.mp4 ! qtdemux ! h264parse ! nvv4l2decoder cudadec-memtype=0 ! fakesink sync=false
Memory is never above 300MB
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 458908 C gst-launch-1.0 309MiB |
Edit: We were able to see that NVDEC produces invalid frames for this stream as well; so it might not be related to specifically deepstream. The deepstream-6.0 machine is at driver version 510.47.03 + cuda 11.4 while the problem machines are at 515.48.07 + cuda 11.6