Decode MJPEG stream in deepstream use GPUs memory continuously increasing

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi, Im working on deepstream app with stream video: mjpeg, yuvj420p. And realized GPUs memory continuously increasing. While read rtsp stream in the same pipeline is stable.

I tested on GeForce GTX 1080 Ti and Tesla P4 too.

I run deepstream app with valgrind tool to check memory leak. And below is some log result.

...
==12685==    by 0x8A78AE1: NvBufSurfaceCreateImpl (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78801: NvBufSurfaceCreate (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0xEFFFD3D: gst_nvinfer_allocator_alloc(_GstAllocator*, unsigned long, _GstAllocationParams*) (in /opt/nvidia/deepstream/deepstream-4.0/lib/gst-plugins/libnvdsgst_infer.so)
==12685==    by 0x4E7A054: gst_buffer_new_allocate (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7DA67: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7DB15: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7E0EB: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7E49F: gst_buffer_pool_set_active (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0xF003AB4: gst_nvinfer_start(_GstBaseTransform*) (in /opt/nvidia/deepstream/deepstream-4.0/lib/gst-plugins/libnvdsgst_infer.so)
==12685==    by 0xBFDC26F: ??? (in /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0.1405.0)
==12685==    by 0xBFDC504: ??? (in /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0.1405.0)
==12685== 
==12685== Conditional jump or move depends on uninitialised value(s)
==12685==    at 0x8A7A2F0: CreateCudaBufferBatch (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78AE1: NvBufSurfaceCreateImpl (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78801: NvBufSurfaceCreate (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0xEFFFD3D: gst_nvinfer_allocator_alloc(_GstAllocator*, unsigned long, _GstAllocationParams*) (in /opt/nvidia/deepstream/deepstream-4.0/lib/gst-plugins/libnvdsgst_infer.so)
==12685==    by 0x4E7A054: gst_buffer_new_allocate (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7DA67: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7DB15: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7E0EB: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E7E49F: gst_buffer_pool_set_active (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0xF003AB4: gst_nvinfer_start(_GstBaseTransform*) (in /opt/nvidia/deepstream/deepstream-4.0/lib/gst-plugins/libnvdsgst_infer.so)
==12685==    by 0xBFDC26F: ??? (in /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0.1405.0)
==12685==    by 0xBFDC504: ??? (in /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0.1405.0)
==12685== 
==12685== Thread 16 multiqueue0:src_:
==12685== Conditional jump or move depends on uninitialised value(s)
==12685==    at 0x8A79DA8: CreateCudaBufferBatch (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78AE1: NvBufSurfaceCreateImpl (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78801: NvBufSurfaceCreate (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0xC4D6DE2D: jpeg_read_raw_data (jdapistd.c:460)
==12685==    by 0xC4B4AAE1: gst_jpeg_dec_decode_direct(_GstJpegDec*, _GstVideoFrame*) (gstjpegdec.c:983)
==12685==    by 0xC4B4BD42: gst_jpeg_dec_handle_frame(_GstVideoDecoder*, _GstVideoCodecFrame*) (gstjpegdec.c:1308)
==12685==    by 0x881CEA0: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x882548E: gst_video_decoder_have_frame (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0xC4B49160: gst_jpeg_dec_parse(_GstVideoDecoder*, _GstVideoCodecFrame*, _GstAdapter*, int) (gstjpegdec.c:519)
==12685==    by 0x881D062: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x881FA2B: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x8820071: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685== 
==12685== Conditional jump or move depends on uninitialised value(s)
==12685==    at 0x8A7A2F0: CreateCudaBufferBatch (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78AE1: NvBufSurfaceCreateImpl (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78801: NvBufSurfaceCreate (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0xC4D6DE2D: jpeg_read_raw_data (jdapistd.c:460)
==12685==    by 0xC4B4AAE1: gst_jpeg_dec_decode_direct(_GstJpegDec*, _GstVideoFrame*) (gstjpegdec.c:983)
==12685==    by 0xC4B4BD42: gst_jpeg_dec_handle_frame(_GstVideoDecoder*, _GstVideoCodecFrame*) (gstjpegdec.c:1308)
==12685==    by 0x881CEA0: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x882548E: gst_video_decoder_have_frame (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0xC4B49160: gst_jpeg_dec_parse(_GstVideoDecoder*, _GstVideoCodecFrame*, _GstAdapter*, int) (gstjpegdec.c:519)
==12685==    by 0x881D062: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x881FA2B: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x8820071: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685== 
==12685== Thread 1:
==12685== Conditional jump or move depends on uninitialised value(s)
==12685==    at 0x8A7A4C2: ReleaseCudaBufferBatch (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78BB9: NvBufSurfaceDestroyImpl (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A7881B: NvBufSurfaceDestroy (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0xC4DA7E33: self_destruct(jpeg_common_struct*) (jmemmgr.c:1033)
==12685==    by 0xC4D66781: jpeg_destroy (jcomapi.c:70)
==12685==    by 0xC4D6A65D: jpeg_destroy_decompress (jdapimin.c:201)
==12685==    by 0xC4B4801D: gst_jpeg_dec_finalize(_GObject*) (gstjpegdec.c:144)
==12685==    by 0x518C011: g_object_unref (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
==12685==    by 0x4EA81A9: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x4E80D76: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==12685==    by 0x5417284: g_main_context_dispatch (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
==12685==    by 0x541764F: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
==12685== 
==12685== Thread 31 multiqueue2:src_:
==12685== Conditional jump or move depends on uninitialised value(s)
==12685==    at 0x8A7A4C2: ReleaseCudaBufferBatch (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A78BB9: NvBufSurfaceDestroyImpl (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0x8A7881B: NvBufSurfaceDestroy (in /opt/nvidia/deepstream/deepstream-4.0/lib/libnvbufsurface.so)
==12685==    by 0xC4D6E1C6: jpeg_read_raw_data (jdapistd.c:511)
==12685==    by 0xC4B4AAE1: gst_jpeg_dec_decode_direct(_GstJpegDec*, _GstVideoFrame*) (gstjpegdec.c:983)
==12685==    by 0xC4B4BD42: gst_jpeg_dec_handle_frame(_GstVideoDecoder*, _GstVideoCodecFrame*) (gstjpegdec.c:1308)
==12685==    by 0x881CEA0: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x882548E: gst_video_decoder_have_frame (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0xC4B49160: gst_jpeg_dec_parse(_GstVideoDecoder*, _GstVideoCodecFrame*, _GstAdapter*, int) (gstjpegdec.c:519)
==12685==    by 0x881D062: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x881FA2B: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
==12685==    by 0x8820071: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1405.0)
...

The deepstream app run and until too get full memory then stuck and hang on. What I can do? thank you for your help!

2

Can you share the config file you used which can repro this issue, also video file to us for further check? and which DS version you are using?

3

Hello amycao,

I am using DS version 4.0.1. I run the config of deepstream-app bellow. And the link of stream you can simulated by git-repo https://github.com/jacksonliam/mjpg-streamer

# Copyright (c) 2018 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=2
uri=...[simulated stream _ 1]
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
nvbuf-memory-type=0

[sink1]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=0
#iframeinterval=10
#use for encode type 3 or 4
bitrate=2000000
output-file=./mjpeg_klt.mp4
#source-id=0

[osd]
enable=1
gpu-id=0
border-width=2
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=3

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=./models/resnet10.caffemodel_b4_int8.engine
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tracker]
enable=1
tracker-width=640
tracker-height=368
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_nvdcf.so
ll-lib-file=./libs/libnvds_mot_klt.so
#[@note] ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

[tests]
file-loop=0
4

Hi,
I am confused about which source type have this issue? first you said
Hi, Im working on deepstream app with stream video: mjpeg, yuvj420p. And realized GPUs memory continuously increasing. While read rtsp stream in the same pipeline is stable.
then
I am using DS version 4.0.1. I run the config of deepstream-app bellow. And the link of stream you can simulated by git-repo GitHub - jacksonliam/mjpg-streamer: Fork of http://sourceforge.net/projects/mjpg-streamer/
seems this kind of stream is like rtsp stream?
so can you confirm which source type have this issue? mjpeg local file or rtsp mjpeg stream?

5

Hi,

Sorry about inconvenience. Source stream is MJPEG with protocol HTTP. when start mjpeg-streamer you can get stream by URI: http://127.0.0.1:8080/?action=stream_0.

6

Will try to repro your issue, and get back to you after.

7

HI
Minor memory leak issue exist, you can see release note, https://docs.nvidia.com/metropolis/deepstream/DeepStream_4.0.2_Release_Notes.pdf
2.0 LIMITATIONS
ď‚„ Small memory leak observed; fixes in progress.
I still have issue to setup streaming by GitHub - jacksonliam/mjpg-streamer: Fork of http://sourceforge.net/projects/mjpg-streamer/
could you please do like below to check the mem leak number(please use config as attached, just need to change source uri)

1 run the sample
2 get the pid
3 run the script ./script name $pid
and lastly can you paste the output here?
source30_1080p_dec_infer-resnet_tiled_display_int8.txt (3.5 KB)
dump_RSS_mem_runningtime_pid.7z (413 Bytes)

8

Hi trild-vietnam,

Could you try the provided steps to check the mem leak number?
Please use config as attached at previous comments, just need to change source uri.

9

Hi trild-vietnam,

We haven’t heard back from you in a couple weeks, so marking this topic closed.
Please open a new forum issue when you are ready and we’ll pick it up there.

10

hello,

Sorry for to be late,
Here is out put affter run script get pip with deepstream app

Getting RSS details for process
1194228, 02:13
Getting RSS details for process
1211504, 03:13
Getting RSS details for process
1210536, 04:13
Getting RSS details for process
1210928, 05:13
Getting RSS details for process
1201592, 06:13
Getting RSS details for process
1206548, 07:13
Getting RSS details for process
1213680, 08:13
Getting RSS details for process
1203640, 09:14
Getting RSS details for process
1211564, 10:14
Getting RSS details for process
1211320, 11:14
Getting RSS details for process
1207740, 12:14
Getting RSS details for process
1209032, 13:14
Getting RSS details for process
1215328, 14:14
Getting RSS details for process
1216408, 15:14
Getting RSS details for process
1210768, 16:14
Getting RSS details for process
1210500, 17:14
Getting RSS details for process
1212536, 18:14
Getting RSS details for process
1205332, 19:14
Getting RSS details for process
1219980, 20:14
Getting RSS details for process
1216620, 21:14
Getting RSS details for process
1215744, 22:14
Getting RSS details for process
927672, 23:14
Getting RSS details for process
933020, 24:16
Getting RSS details for process

Note: GPUs memory still continuously increasing affter 5 minutes is full. Only memory allocated for engine is keep stable. No more other application use GPUs memory. Affter about 20 minutes app was killed.

12

Could you please share output when run the script and also the output of the following commands when the pipeline is running, would be useful to understand the setup and decoder/gpu utilization.

nvidia-smi
nvidia-smi -q -d UTILIZATION -l 1 -i “gpuid”

13

Hello amycao, bellow is log file when use

nvidia-smi -q -d UTILIZATION -l 1 -i “gpuid”

14

Could you please specify the detailed repro steps for a further check?
→ see it.

15

Sorry, but see what?

16

How to create http mjpeg stream.

17

Hello,
you can get this github repos : GitHub - ledinhtri97/mjpg-streamer: forked from jacksonliam / mjpg-streamer

Following bellow step:
1: Create folder saves images extract: /home/path-to-folder:
2: Run ffmpeg extract one video file to sequences images:

ffmpeg -i video-file.mp4 -vf fps=0.2 -t 600 /home/path-to-folder/image-%06d.jpg

3: Run ./mjpg_streamer in repos github:

./mjpg_streamer -o “output_http.so -w ./www -l 0.0.0.0 -p 8283” -i “input_file.so -d 0.01 -f /home/path-to-folder”

Now you can get link mjpeg.

18

Hello amycao, is my answer setup simulated mjpeg stream worked? Are there any problems?

19

We will fix it in upcoming release, thanks for patience.

1 Like