Issues with Deepstream SDK (deepstream-nvdsanalytics)- Memory Usage and Kafka Broker

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

• Hardware Platform (Jetson AGX Xavier)
• DeepStream 5.0
• JetPack Version 4.4
• TensorRT Version 7.10
• Issue Type( Bugs)

I’m running deepstream-nvdsanalytics-test app for 24 hours and encounter some bugs or issues.
Fyi, I modified the deepstream app to be able to send message using nvmsgconv plugin. I chose kafka as the message broker. I’m using IP cam as the video source through RTSP stream. I monitored the received messages on the grafana dashboard.

The first issue is related to memory, there’s a constant increase in memory usage (up to 100%) when the app is running. I read that it might be related to the RTSP stream, but is there any update about it?

The second issue or bug is, at a certain time (after running for 10-12 hours), the deepsream pipeline produced the same detection value (kafka keep receiving same value). I’ve checked either the kafka producer and my kafka client, and could not find issues related with them. Hence, the issues might be related to deepstream pipeline.

If necessary I could send my code so that you could produce the same issues.

thanks.

#3

For these issues, can you reproduce by the original deepstream-nvdsanalytics-test?

#4

I have not tried it yet, but I assume it would be the same since I didn’t change the original code. I only added lines for the messaging plugin.
Also it’s bit hard to debug the original code since currently the jetson unit has been deployed on edge, and I can only and control it via ssh.
The only way I can monitor the performance by a look at the messages received on my dashboard.

#5

Can you provide steps and files needed of reproducing the problem?

#6

deepstream_deepstream-app_deepstream_nvdsanalytics.cpp (56.2 KB)

this is our deepstream-app file.
The step is pretty similar with deepstream-test4

this file is running on background as a service file.

#7

Will try to repro the issue, and get back to you.

#8

Hi,
I before run test5 sample with local video, but without memory leak, will repro with rtsp source, btw, it would be better you can test on your side using your product software with local video if have the issue. here topic for your reference, OOM of kafka? - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

The second issue or bug is, at a certain time (after running for 10-12 hours), the deepsream pipeline produced the same detection value (kafka keep receiving same value).

–> can you specify what kind of value? better you can paste some clip log.

#11

I run with test5 sample which demonstrates message sending, using rtsp camera source, but no memory leak from test5 sample, using script dump_RSS_mem_runningtime_pid.sh.log (365 Bytes)
to capture real memory loaded into ram, run around 3 hours, RSS stable at 3600,
nvidia@nvidia-desktop:~$ sudo ./dump_RSS_mem_runningtime_pid.sh 9083
[sudo] password for nvidia:
Getting RSS details for process
3600, 00:15
Getting RSS details for process
3600, 01:16
Getting RSS details for process
3600, 02:16
Getting RSS details for process
3600, 03:16
Getting RSS details for process
3600, 04:16
Getting RSS details for process
3600, 05:16
Getting RSS details for process
3600, 06:16
Getting RSS details for process
3600, 07:16
Getting RSS details for process
3600, 08:16
Getting RSS details for process
3600, 09:16
Getting RSS details for process
3600, 10:16
Getting RSS details for process
3600, 11:16
Getting RSS details for process
3600, 12:16
Getting RSS details for process
3600, 13:16
Getting RSS details for process
3600, 14:16
Getting RSS details for process
3600, 15:17
Getting RSS details for process
3600, 16:17
Getting RSS details for process
3600, 17:17
Getting RSS details for process
3600, 18:17
Getting RSS details for process
3600, 19:17
Getting RSS details for process
3600, 20:17
Getting RSS details for process
3600, 21:17
Getting RSS details for process
3600, 22:17
Getting RSS details for process
3600, 23:17
Getting RSS details for process
3600, 24:17
Getting RSS details for process
3600, 25:17
Getting RSS details for process
3600, 26:17
Getting RSS details for process
3600, 27:17
Getting RSS details for process
3600, 28:18
Getting RSS details for process
3600, 29:18
Getting RSS details for process
3600, 30:18
Getting RSS details for process
3600, 31:18
Getting RSS details for process
3600, 32:18
Getting RSS details for process
3600, 33:18
Getting RSS details for process
3600, 34:18
Getting RSS details for process
3600, 35:18
Getting RSS details for process
3600, 36:18
Getting RSS details for process
3600, 37:18
Getting RSS details for process
3600, 38:18
Getting RSS details for process
3600, 39:18
Getting RSS details for process
3600, 40:18
Getting RSS details for process
3600, 41:18
Getting RSS details for process
3600, 42:19
Getting RSS details for process
3600, 43:19
Getting RSS details for process
3600, 44:19
Getting RSS details for process
3600, 45:19
Getting RSS details for process
3600, 46:19
Getting RSS details for process
3600, 47:19
Getting RSS details for process
3600, 48:19
Getting RSS details for process
3600, 49:19
Getting RSS details for process
3600, 50:19
Getting RSS details for process
3600, 51:19
Getting RSS details for process
3600, 52:19
Getting RSS details for process
3600, 53:19
Getting RSS details for process
3600, 54:19
Getting RSS details for process
3600, 55:20
Getting RSS details for process
3600, 56:20
Getting RSS details for process
3600, 57:20
Getting RSS details for process
3600, 58:20
Getting RSS details for process
3600, 59:20
Getting RSS details for process
3600, 01:00:20
Getting RSS details for process
3600, 01:01:20
Getting RSS details for process
3600, 01:02:20
Getting RSS details for process
3600, 01:03:20
Getting RSS details for process
3600, 01:04:20
Getting RSS details for process
3600, 01:05:20
Getting RSS details for process
3600, 01:06:20
Getting RSS details for process
3600, 01:07:20
Getting RSS details for process
3600, 01:08:20
Getting RSS details for process
3600, 01:09:21
Getting RSS details for process
3600, 01:10:21
Getting RSS details for process
3600, 01:11:21
Getting RSS details for process
3600, 01:12:21
Getting RSS details for process
3600, 01:13:21
Getting RSS details for process
3600, 01:14:21
Getting RSS details for process
3600, 01:15:21
Getting RSS details for process
3600, 01:16:21
Getting RSS details for process
3600, 01:17:21
Getting RSS details for process
3600, 01:18:21
Getting RSS details for process
3600, 01:19:21
Getting RSS details for process
3600, 01:20:21
Getting RSS details for process
3600, 01:21:21
Getting RSS details for process
3600, 01:22:22
Getting RSS details for process
3600, 01:23:22
Getting RSS details for process
3600, 01:24:22
Getting RSS details for process
3600, 01:25:22
Getting RSS details for process
3600, 01:26:22
Getting RSS details for process
3600, 01:27:22
Getting RSS details for process
3600, 01:28:22
Getting RSS details for process
3600, 01:29:22
Getting RSS details for process
3600, 01:30:22
Getting RSS details for process
3600, 01:31:22
Getting RSS details for process
3600, 01:32:22
Getting RSS details for process
3600, 01:33:22
Getting RSS details for process
3600, 01:34:22
Getting RSS details for process
3600, 01:35:22
Getting RSS details for process
3600, 01:36:23
Getting RSS details for process
3600, 01:37:23
Getting RSS details for process
3600, 01:38:23
Getting RSS details for process
3600, 01:39:23
Getting RSS details for process
3600, 01:40:23
Getting RSS details for process
3600, 01:41:23
Getting RSS details for process
3600, 01:42:23
Getting RSS details for process
3600, 01:43:23
Getting RSS details for process
3600, 01:44:23
Getting RSS details for process
3600, 01:45:23
Getting RSS details for process
3600, 01:46:23
Getting RSS details for process
3600, 01:47:23
Getting RSS details for process
3600, 01:48:23
Getting RSS details for process
3600, 01:49:24
Getting RSS details for process
3600, 01:50:24
Getting RSS details for process
3600, 01:51:24
Getting RSS details for process
3600, 01:52:24
Getting RSS details for process
3600, 01:53:24
Getting RSS details for process
3600, 01:54:24
Getting RSS details for process
3600, 01:55:24
Getting RSS details for process
3600, 01:56:24
Getting RSS details for process
3600, 01:57:24
Getting RSS details for process
3600, 01:58:24
Getting RSS details for process
3600, 01:59:24
Getting RSS details for process
3600, 02:00:24
Getting RSS details for process
3600, 02:01:24
Getting RSS details for process
3600, 02:02:25
Getting RSS details for process
3600, 02:03:25
Getting RSS details for process
3600, 02:04:25
Getting RSS details for process
3600, 02:05:25
Getting RSS details for process
3600, 02:06:25
Getting RSS details for process
3600, 02:07:25
Getting RSS details for process
3600, 02:08:25
Getting RSS details for process
3600, 02:09:25
Getting RSS details for process
3600, 02:10:25
Getting RSS details for process
3600, 02:11:25
Getting RSS details for process
3600, 02:12:25
Getting RSS details for process
3600, 02:13:25
Getting RSS details for process
3600, 02:14:25
Getting RSS details for process
3600, 02:15:25
Getting RSS details for process
3600, 02:16:26
Getting RSS details for process
3600, 02:17:26
Getting RSS details for process
3600, 02:18:26
Getting RSS details for process
3600, 02:19:26
Getting RSS details for process
3600, 02:20:26
Getting RSS details for process
3600, 02:21:26
Getting RSS details for process
3600, 02:22:26
Getting RSS details for process
3600, 02:23:26
Getting RSS details for process
3600, 02:24:26
Getting RSS details for process
3600, 02:25:26
Getting RSS details for process
3600, 02:26:26
Getting RSS details for process
3600, 02:27:26
Getting RSS details for process
3600, 02:28:26
Getting RSS details for process
3600, 02:29:27
Getting RSS details for process
3600, 02:30:27
Getting RSS details for process
3600, 02:31:27
Getting RSS details for process
3600, 02:32:27
Getting RSS details for process
3600, 02:33:27
Getting RSS details for process
3600, 02:34:27
Getting RSS details for process
3600, 02:35:27
Getting RSS details for process
3600, 02:36:27
Getting RSS details for process
3600, 02:37:27
Getting RSS details for process
3600, 02:38:27
Getting RSS details for process
3600, 02:39:27
Getting RSS details for process
3600, 02:40:27
Getting RSS details for process
3600, 02:41:27
Getting RSS details for process
3600, 02:42:27
Getting RSS details for process
3600, 02:43:28
Getting RSS details for process
3600, 02:44:28
Getting RSS details for process
3600, 02:45:28
Getting RSS details for process
3600, 02:46:28
Getting RSS details for process
3600, 02:47:28
Getting RSS details for process
3600, 02:48:28
Getting RSS details for process
3600, 02:49:28
Getting RSS details for process
3600, 02:50:28
Getting RSS details for process
3600, 02:51:28
Getting RSS details for process
3600, 02:52:28
Getting RSS details for process
3600, 02:53:28
Getting RSS details for process
3600, 02:54:28
Getting RSS details for process
3600, 02:55:28
Getting RSS details for process
3600, 02:56:29
Getting RSS details for process
3600, 02:57:29
Getting RSS details for process
3600, 02:58:29
Getting RSS details for process
3600, 02:59:29
Getting RSS details for process
3600, 03:00:29
Getting RSS details for process
3600, 03:01:29
Getting RSS details for process
3600, 03:02:29
Getting RSS details for process
3600, 03:03:29
Getting RSS details for process
3600, 03:04:29
Getting RSS details for process
3600, 03:05:29
Getting RSS details for process
3600, 03:06:29
Getting RSS details for process
3600, 03:07:29

and java used memory increased, which used by kafka, kafka broker running on one x86 server.
first capture when run at the beginning,

top - 17:17:39 up 23 days, 6:09, 4 users, load average: 0.00, 0.03, 0.06
Tasks: 286 total, 1 running, 203 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.4 us, 0.1 sy, 0.0 ni, 99.4 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16102280 total, 4296244 free, 1821112 used, 9984924 buff/cache
KiB Swap: 31624188 total, 31622140 free, 2048 used. 13973132 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2292 rabbitmq 20 0 5358904 63988 6200 S 2.0 0.4 924:12.45 beam.smp
2327 tse 20 0 8003476 829512 27992 S 0.7 5.2 1:30.46 java
8899 tse 20 0 3673488 158080 27704 S 0.3 1.0 0:09.46 java
9324 tse 20 0 43692 4164 3456 R 0.3 0.0 0:00.04 top

last capture when the sample stop running.

top - 20:19:21 up 23 days, 9:11, 2 users, load average: 0.00, 0.02, 0.00
Tasks: 268 total, 1 running, 186 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.6 us, 0.1 sy, 0.0 ni, 99.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16102280 total, 4460352 free, 2123000 used, 9518928 buff/cache
KiB Swap: 31624188 total, 31622140 free, 2048 used. 13669292 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2292 rabbitmq 20 0 5358904 63988 6200 S 5.0 0.4 929:29.86 beam.smp
2327 tse 20 0 8003476 0.984g 28012 S 0.7 6.4 2:24.42 java
9324 tse 20 0 43692 4164 3456 R 0.3 0.0 1:06.97 top
25584 root 20 0 0 0 0 I 0.3 0.0 0:44.98 kworker/0:1
1 root 20 0 226224 9776 6624 S 0.0 0.1 0:27.54 systemd

The first issue is related to memory, there’s a constant increase in memory usage (up to 100%) when the app is running.

–> the issue you mentioned is like above mentioned? if yes, This is something beyond the scope of deepstream. not sure if installing the latest kafka package helps solve the issue. if not, please specify details.

1 Like
#13