After 10h or more runtime, swap and memory is suddenly full used on jestson nano?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

To save memory I disabled graphical interface so for a long time all is ok (1,5G ram free and swap is around 380 mega), but after 10 or sometime more it depends, suddenly the jetson nano sees all its swap and memory used ? and finally the kernel kills deepstram-app process ? swap and ram is back to normal.

I have seen that once on TX2 also? I have modified deepstream-app to save continuously multiple files on a nvme drive.
I am building a smart DVR for home automation I am using production version nano on an auvidia carrier.

Best Regards,

Laurent

480Ă—640 470 KB 

# Copyright (c) 2019 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=2
columns=2
#width=1280
#height=720
width=1920
height=1080
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP //porte garage cam
type=4
uri=rtsp://admin:xxxx@192.168.1.252:554/Streaming/Channels/1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
rtsp-reconnect-interval-sec=10
latency=200

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP //cuisine cam
type=4
uri=rtsp://admin:xxxx@192.168.1.244:554/h264Preview_01_main
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
rtsp-reconnect-interval-sec=10
latency=200

[source2]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP //jardin cam
type=4
uri=rtsp://admin:Gimtf7am94@192.168.1.247/Streaming/Channels/3
#num-sources=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
rtsp-reconnect-interval-sec=10
latency=200

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP // porte cam
type=4
uri=rtsp://admin:wifibot78@192.168.1.248:554/1/h264major
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
rtsp-reconnect-interval-sec=10
latency=200

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
qos=0
nvbuf-memory-type=0
overlay-id=1

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=1
#iframeinterval=10
bitrate=3000000
output-file=/mnt/nvme/videos/camia-%s-%03d.mp4

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
sync=0
nvbuf-memory-type=0
bitrate=2000000
#iframeinterval=10
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=400000
## Set muxer output width and height
width=1920
height=1080

##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_fp16.engine
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=4
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_nano.txt

[tracker]
enable=1
tracker-width=480
tracker-height=272
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#ll-config-file required for IOU only
#ll-config-file=iou_config.txt
gpu-id=0

[tests]
file-loop=0
2

finally when i disable filesink (in my case I use splitmuxsink) , after 2 days all ok, no memory leak ? So it seems that filesinks consume RAM when used for a long time ?

this could be related to gstreamer in general ?

4

Hi,
Did you check the disk usage when the issue happened? I guess, when the issue happened, disk was full, and the more generated data had to be put in memory and then swap, which cause memory and swap became full quickly.

Thanks!

5

Hello @laurentj6jzo, I have this issues too when using multifilesink. Just leak swap memory but make slow down performance deepstream. Did you resolved or have any suggest solution ?

6

Hello,

Sorry I did not see your answer.

No I had only 33% drive used when the problem occurs. I have a 500G nvme hd only for recording mp4 videos.

I thought it was because sometimes it does not close well the file, so I backported to gstreamer 1.14.5 splitmuxsink properties “async-finalize” from 1.16 version : same problem
after a longue time.

Could it be a problem related to this on the Deepstream FAQ ?:

Fix for a memory accumulation bug in GstBaseParse
A memory accumulation bug was found in GStreamer’s Base Parse class which potentially affects all codec parsers provided by GStreamer. This bug is seen only with long duration seekable streams (mostly containerized files e.g. mp4). This does not affect live sources like RTSP.

or something related to the mp4 muxer building the index table in memory, before writing it out to disk on eos ?

Strange think is that this memory leak happens suddenly not increasing regularly during runtime …

Thank you for your response.

Laurent

7

hello,

not yet solved, I have a few clues, just see my last post

Laurent

8

Hi laurentj6jzo,
Sorry for late reply!
Did you solve this issue with the Fix for a memory accumulation bug ?

Besides the memory issue, from your screenshot, I have one question, why do you start so many “deepstream-app”? As you may know, one deepstream process can handle many
video input streams, even the stream types are different.
And, as you can see in your screenshot, each deepstream-app consumes large of mmeory. Since each deepstream app will have its own CUDA/cuDNN/TensorRT/Decode contextes, which not only consumes more memory but also reduces the processing efficiency.

So, I think, you should firstly try to use one deepstream process to handle these streams, this must help save the memory usage.

9

And, for the original swap memory issue, as you found, it’s related to the frequent file writing. In Linux OS, to improve disk WRITE performance, it normally cache the WRITE data in memory firstly, when memory is shortage, the data may be put in swap. You could find many such articles in Network, e.g. filesystems - Can I configure my Linux system for more aggressive file system caching? - Unix & Linux Stack Exchange. You could find some solution in the article.

1 Like
11

Hello,

Those many process seems normal, it is a way HTOP displays some forks I thinks ? Original deepstream-app does the same with HTOP and 4 rtsp live stream.
In fact, the problems seem to be related to PTS, not those many process or filesystem.
In fact each time the RAM suddenly grows up, the file were not splited well, next file was smaller and the next one was more than what I told splitmuxsink to split?

-rw-rw-r-- 1 domo domo 336650167 avril 1 11:04 camia-Apr01-2020-10-49-52-099.mp4
-rw-rw-r-- 1 domo domo 336019592 avril 1 11:19 camia-Apr01-2020-11-04-54-100.mp4
-rw-rw-r-- 1 domo domo 288521415 avril 1 12:22 camia-Apr01-2020-11-19-57-101.mp4
-rw-rw-r-- 1 domo domo 1114375048 avril 1 12:22 camia-Apr01-2020-12-22-41-102.mp4*

So I focused on something related to timestamps:
Since the beginning Deepstream-app was not working with one of my cameras, it was throwing:

Debug info: gstqtmux.c(4561): gst_qt_mux_add_buffer (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin2/GstSplitMuxSink:sink_sub_bin_sink2/GstMP4Mux:muxer:
Buffer has no PTS.
Quitting

To make it works, I used live-source=1 and were obliged to use in my code :
“gst_base_parse_set_pts_interpolation” to interpolate PTS, and it was ok, no error except for sudden memory leak after 10 or sometimes 17 hours.

So I decided yesterday to take off gst_base_parse_set_pts_interpolation function and set live-source=0 (whereas I am working with live sources) so in this case the muxer calculates timestamps based on the frame rate of the source which first negotiates capabilities with the muxer.
And now all is ok? the DVR had runs for more than 24 hours without any problem and the RAM is stable … I will continue to make more tests