Jetson Nano shows 100% CPU Usage after 30 minutes with Deepstream-app demo

Hi,

Thanks very much for trying to repeat this.

I am using the original Jetson Nano Developer kit with a 64GB Class 10 sdcard (no eMMC).
I created my sdcard image from your latest updates on 9/24/19…Jetpack 4.2.2 and Deepstream 4.0.1.

Did you make the changes that I suggested - especially the following?

–In [sink0],“sync” was changed from 1 to 0
drop-frame-interval=3

From your 30FPS performance, I am guessing that you don’t have the drop-frame-interval enabled to 3.

Anyhow, I power cycled my Nano this morning & decided to repeat the experiment but I am using the configuration file that I attached yesterday - that includes the 2 changes above.
First of all I ran the two commands you suggested:

hduser@NVNano4:~/LENS$ sudo nvpmodel -m 0
hduser@NVNano4:~/LENS$ sudo jetson_clocks
hduser@NVNano4:~/LENS$ date
Thu Oct 24 08:27:59 MDT 2019

Everything started off ok & Performance was as expected at approx. 10 FPS (because of drop-frame-interval=3).

**PERF: 10.66 (11.25)   16.46 (16.33)   18.40 (15.31)   17.19 (12.46)   8.29 (11.83)       16.26 (16.20)   7.54 (10.66)    9.43 (9.83)
**PERF: 11.59 (11.27)   15.10 (16.24)   14.33 (15.24)   16.92 (12.79)   8.51 (11.59)       15.64 (16.16)   7.63 (10.44)    13.84 (10.12)

I will use the same utility that you used - tegrastats. As shown, the CPUs started off ok also.

hduser@NVNano4:~/LENS$ sudo tegrastats
RAM 3160/3956MB (lfb 110x4MB) SWAP 62/1978MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [26%@1428,26%@1428,29%@1428,27%@1428] EMC_FREQ 39%@1600 GR3D_FREQ 0%@921 NVDEC 716 APE 25 PLL@26C CPU@29.5C PMIC@100C GPU@27.5C AO@34C thermal@28.75C POM_5V_IN 4971/4           971 POM_5V_GPU 328/328 POM_5V_CPU 820/820
RAM 3160/3956MB (lfb 110x4MB) SWAP 62/1978MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [29%@1428,28%@1428,25%@1428,27%@1428] EMC_FREQ 39%@1600 GR3D_FREQ 99%@921 NVDEC 716 APE 25 PLL@26C CPU@29.5C PMIC@100C GPU@28C AO@34C thermal@28.75C POM_5V_IN 6279/56           25 POM_5V_GPU 1779/1053 POM_5V_CPU 728/774

After a few minutes, the CPU usage started going up & FPS going down.

In <30 minutes, the Performance had dramatically dropped as shown:

hduser@NVNano4:~/LENS$ date;tail log_ds4_wip.log
Thu Oct 24 08:54:33 MDT 2019
**PERF: 5.04 (8.58)     5.57 (8.62)     4.72 (8.68)     5.40 (8.59)     5.40 (8.49)5.11 (8.68)     4.80 (8.67)     4.80 (8.59)

Tegrastats confirm that the 4 CPUs are pegged at 100%:

hduser@NVNano4:~/LENS$ date;sudo tegrastats --interval 5000
Thu Oct 24 08:55:53 MDT 2019
RAM 3295/3956MB (lfb 107x4MB) SWAP 168/1978MB (cached 3MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1428,100%@1428,100%@1428,100%@1428] EMC_FREQ 27%@1600 GR3D_FREQ 0%@921 NVDEC 716 APE 25 PLL@30C CPU@33C PMIC@100C GPU@31.5C AO@38C thermal@32.25C POM_5V_IN 6033/6033 POM_5V_GPU 815/815 POM_5V_CPU 1510/1510
RAM 3290/3956MB (lfb 107x4MB) SWAP 175/1978MB (cached 3MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1428,100%@1428,100%@1428,100%@1428] EMC_FREQ 27%@1600 GR3D_FREQ 0%@921 NVDEC 716 APE 25 PLL@30C CPU@33.5C PMIC@100C GPU@31.5C AO@38C thermal@32.5C POM_5V_IN 5879/5956 POM_5V_GPU 694/754 POM_5V_CPU 1469/1489

Please let me know if you need any more details.

Thanks,
Vince

Hi,

this issue maybe the same with me.

[url]https://devtalk.nvidia.com/default/topic/1065188/deepstream-sdk/why-nvv4l2decoder-use-too-much-cpu-/post/5395132/#5395132[/url]

Thanks.

Hi ClancyLian,

Thanks for your update.

So I read your post & my take-away from it is that you had normal performance if you don’t set the drop-frame-interval attribute.
Since you are using a different SOC to me… “tesla P4 gpu” & we have narrowed down our extremely high use of CPUs to the one variable, it sure smells like a bug to me.

Hopefully NVIDIA will give some priority to this issue & provide a solution!

Thanks,
Vince

Hi vincent.mcgarry,

We will investigate this issue and update to you. Thanks!

hi, Vince , carolyuu

Is there a temporary solution to drop frame before decoder ?

Thanks.

Hi ClancyLian,

Our internal team still investigate issue.
Sorry for late responds and inconvenience.

Hi,

I have had the same problem for several days. I set “drop-frame-interval” too, and the cpu increase over time.

Has this issue been solved?

I also meet the issue in Jetson tx2. once enable drop-frame-interval, the problem will happen in one hour. Jetpack 4.2.2 and deepstream 4.0.1. Can you help me? my project will deploy in December.

Hi,

Does anyone from Nvidia have any update on this issue?

It has been over a month & it appears from this thread that several others are experiencing the same issue.

Thanks,
Vince

Hi
we can recreated your issue, Our dev team is looking into this issue, sorry for the late response.

Hi,
Please apply the attached patch to gst-v4l2, rebuild and replace libgstnvvideo4linux2.so

gst-v4l2source is in
https://developer.nvidia.com/embedded/r32-2-3_Release_v1.0/Sources/T186/public_sources.tbz2
https://developer.nvidia.com/embedded/r32-2-3_Release_v1.0/Sources/T210/public_sources.tbz2
DS401_JETSON_TEST_0001-gstv4l2dec-Fix-high-CPU-usage-in-drop-frame.zip (1.08 KB)

Hi, DaneLLL

Have any solution for T4 and V100?

Hi,

You can also build it for x86 PC with NVIDIA dGPU. Please check information about gst-v4l2:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html
Plugin and Library Source Details

Hi DaneLLL,

Thanks very much for the fix to my issue.
I followed your instructions & applied the patch to my Jetson Nano test.

Ran the deepstream-app demo test for at least 16 hours without any failure or increase in CPU usage (actually using 8 internet cameras).
The tegrastats utility shows the following:

RAM 3094/3956MB (lfb 105x4MB) SWAP 75/1978MB (cached 4MB) IRAM 0/252kB(lfb 252kB) CPU [14%@1428,10%@1428,8%@1428,11%@1428] EMC_FREQ 19%@1600 GR3D_FREQ 0%@921 NVDEC 716 APE 25 PLL@29.5C CPU@33C PMIC@100C GPU@31.5C AO@36.5C thermal@32.75C POM_5V_IN 3886/3844 POM_5V_GPU 250/208 POM_5V_CPU 500/521

(Previously the 4 cores would be pegged at 100% within 30 minutes).

Do you know when this “fix” will be formally released?

Thanks,
Vince

Hi vincent, can you share the so file? I have no envirment to build it. thanks very much.

Hi Mike_deng,

I actually built it on the Nano using the instructions from Post#13 (Used T210 source for the Nano).

Since I couldn’t attach the .so, I have compressed it & attached.
(Initially I changed the name & compressed to a .zip, but when attached, the forum detected it as “Infected”. So I compressed to a .7z without changing name & it was accepted)

Thanks,
Vince
libgstnvvideo4linux2.7z (67.6 KB)

To vincent, thanks for your share.

Hi,
Patch in #13 is a quick fix. We have reviewed and updated to new patch set. Please check the attachment. Thanks.
DS40X_TEST_0001-gstv4l2dec-Fix-high-CPU-usage-in-drop-frame.zip (1.87 KB)

1 Like

@DaneLLL
Hi DaneLLL
I repeate this issue with deepstream4.0.1 on T4. After I patch #13, the SM usage is increased to 100%, and the fps decreased. while SM usage is abount 60%~70% before I update the patch.

BTW, Do the following two pipelines have the same performance?

gst-launch-1.0 -e rtmpsrc location="xxxxxxxx" ! typefind ! flvdemux ! h264parse ! nvv4l2decoder drop-frame-interval=25 ! nvvideoconvert ! video/x-raw,format=BGRx ! perf ! fakesink sync=true

gst-launch-1.0 -e rtmpsrc location="xxxxxxxx" ! typefind ! flvdemux ! h264parse ! nvv4l2decoder drop-frame-interval=0 ! videorate ! "video/x-raw(memory:NVMM),framerate=1/1" ! nvvideoconvert ! video/x-raw,format=BGRx ! perf ! fakesink sync=true

I look forward to hearing from you.