Why is the CPU usage so high?

wxsrite · November 29, 2024, 5:47am

Hi All:
Could you please help me figure out why the CPU usage is so high? Is it because there are too many invalid data frames and vi can’t handle them all?

robot@tegra:~$ top
top - 13:36:53 up 1 day,  5:40, 20 users,  load average: 13.08, 13.31, 13.91
Tasks: 415 total,   3 running, 411 sleeping,   0 stopped,   1 zombie
%Cpu(s): 25.8 us, 22.7 sy,  0.0 ni, 48.9 id,  0.6 wa,  1.2 hi,  0.8 si,  0.0 st
MiB Mem :  15519.3 total,    146.0 free,   8137.2 used,   7236.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5176.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5446 root      20   0       0      0      0 R 100.0   0.0   1089:10 vi-output, ar02
   5897 robot     20   0   20.3g   2.4g   1.1g S  60.3  15.6   1083:42 perception-map
   4624 robot     20   0 2838672 623176 413224 S  40.1   3.9 846:27.86 mininav-vio
   4880 robot     20   0   20.1g   1.9g 313564 S  36.1  12.6 666:20.18 camera-node
   4630 robot     20   0 2783444 423116  37856 S  22.5   2.7 446:25.04 mininav-slam
   4622 robot     20   0 4252808 691748  32636 S  21.5   4.4   1649:26 mininav-navigat
   5810 robot     20   0 2969744  49620  18276 S  15.2   0.3 288:44.34 ros2
   4957 root      20   0 1888460  36408  22188 S  14.6   0.2 266:20.67 performance_mon
   4558 robot     20   0 6124640 135088  27384 S  11.3   0.9 223:58.93 mininav-sdk
   4623 robot     10 -10 2195228  76860  14476 S  10.3   0.5 207:31.24 sensorhub_node
   4621 robot     15  -5 2183800  49432  14724 S   7.6   0.3 165:15.91 can_node
3706740 robot     20   0    7804   1856   1428 R   7.0   0.0   0:00.21 du
   4752 root      20   0   23248  14424   5856 S   6.3   0.1 110:32.63 iotop
   1769 robot     20   0   17.2g 149528  27916 S   2.0   0.9  52:00.20 node
   4590 robot     20   0 4751648  62976  16976 S   1.7   0.4  43:52.85 mininav-monitor
   4736 robot     20   0   11976   3460   2472 S   1.7   0.0  30:03.55 top
   4491 robot     20   0  538476  20440   6208 S   1.3   0.1  18:43.04 python3
   5804 robot     20   0 2966760  45668  18416 S   1.3   0.3  19:37.34 ros2
   5807 robot     20   0 2966788  47924  18100 S   1.0   0.3  18:01.06 ros2
3673718 root      20   0       0      0      0 I   1.0   0.0   0:02.04 kworker/u16:2-events_unbound
3706100 root       0 -20       0      0      0 I   1.0   0.0   0:00.15 kworker/u17:0-uvcvideo
    114 root      20   0       0      0      0 S   0.7   0.0  12:19.87 kswapd0
   1227 root      20   0  276948  44352  11828 S   0.7   0.3  17:41.10 network_service
   4465 robot     20   0   12448   6184   2580 S   0.7   0.0   3:09.09 bash
      1 root      20   0  169788  11076   6888 S   0.3   0.1   1:10.51 systemd
     12 root      20   0       0      0      0 S   0.3   0.0   2:11.14 ksoftirqd/0
     13 root      20   0       0      0      0 I   0.3   0.0   0:51.56 rcu_preempt
     49 root      20   0       0      0      0 S   0.3   0.0   1:22.05 ksoftirqd/7
    156 root     -51   0       0      0      0 S   0.3   0.0   4:57.34 irq/135-host_st
    255 root       0 -20       0      0      0 I   0.3   0.0   0:14.95 kworker/7:1H-kblockd
    529 root     -51   0       0      0      0 S   0.3   0.0   0:08.07 irq/232-gk20a_s
    874 root      20   0    2472   1676   1464 S   0.3   0.0   0:18.06 nvfancontrol
   1330 root      20   0    2144   1368   1272 S   0.3   0.0   8:45.61 ksz9897_diag
   2305 proxy     20   0   68168  25112  11940 S   0.3   0.2   0:44.26 squid
   4039 robot     20   0   19944  12736   2044 S   0.3   0.1   2:49.79 tmux: server
   4209 root      20   0    9272   5788   4796 S   0.3   0.0   8:00.03 openvpn
   4703 robot     20   0    7656    444    380 S   0.3   0.0   0:07.97 cat
   4744 robot     20   0   10276   2500   2204 S   0.3   0.0   0:33.69 vmstat
   4973 robot     20   0 2102372   1.8g   1.8g S   0.3  12.1   6:14.80 iox-roudi
3540179 root      20   0       0      0      0 I   0.3   0.0   0:16.99 kworker/4:1-read_counters_wq
3634038 root      20   0       0      0      0 I   0.3   0.0   0:06.39 kworker/u16:1-host_low_prio_wq
3692395 root      20   0       0      0      0 I   0.3   0.0   0:02.02 kworker/0:2-events
3706541 robot     20   0   11952   3684   2696 R   0.3   0.0   0:00.03 top
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.09 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
      9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_kthre
     10 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
     11 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
robot@tegra:~$ sudo perf record -p 5446 -g -- sleep 10
[ perf record: Woken up 13 times to write data ]
[ perf record: Captured and wrote 3.074 MB perf.data (40080 samples) ]
robot@tegra:~$ sudo perf report -i perf.data > perf_report.txt
robot@tegra:~$ chmod 777 perf_report.txt



robot@tegra:~$ top
top - 13:40:28 up 1 day,  5:44, 20 users,  load average: 11.75, 12.71, 13.58
Tasks: 412 total,   2 running, 409 sleeping,   0 stopped,   1 zombie
%Cpu(s): 24.8 us, 20.8 sy,  0.0 ni, 47.3 id,  5.1 wa,  1.2 hi,  0.7 si,  0.0 st
MiB Mem :  15519.3 total,    159.5 free,   8111.6 used,   7248.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5225.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5446 root      20   0       0      0      0 R  12.5   0.0   1092:44 vi-output, ar02     ---->  shift + I
   5897 robot     20   0   20.3g   2.3g   1.1g S   7.6  15.3   1085:53 perception-map
   4624 robot     20   0 2838672 631380 413012 S   5.2   4.0 847:54.71 mininav-vio
   4880 robot     20   0   20.1g   1.9g 313356 S   4.6  12.6 667:37.94 camera-node
   4622 robot     20   0 4252808 689624  32424 S   2.8   4.3   1650:14 mininav-navigat
   4630 robot     20   0 2783444 417020  37644 S   2.7   2.6 447:13.86 mininav-slam
   4957 root      20   0 1888460  36408  22188 S   1.9   0.2 266:52.03 performance_mon
   5810 robot     20   0 2969744  49620  18276 S   1.9   0.3 289:17.36 ros2
   4623 robot     10 -10 2195228  76692  14264 S   1.4   0.5 207:55.21 sensorhub_node
   4558 robot     20   0 6124436 135080  27376 S   1.4   0.8 224:23.19 mininav-sdk
   4621 robot     15  -5 2183800  48804  14468 S   1.0   0.3 165:32.74 can_node
   4752 root      20   0   23248  14424   5856 S   0.7   0.1 110:45.55 iotop
   4590 robot     20   0 4751648  62976  16976 S   0.2   0.4  43:57.91 mininav-monitor
   4736 robot     20   0   11976   3460   2472 S   0.2   0.0  30:07.12 top
   1769 robot     20   0   17.2g 148848  27916 S   0.2   0.9  52:04.97 node
   1227 root      20   0  276948  44352  11828 S   0.1   0.3  17:43.12 network_service
   4491 robot     20   0  538476  20440   6208 S   0.1   0.1  18:45.19 python3
   5804 robot     20   0 2966760  45172  18416 S   0.1   0.3  19:39.67 ros2
3634641 root       0 -20       0      0      0 I   0.1   0.0   0:05.68 kworker/u17:3-uvcvideo
   5807 robot     20   0 2966788  47132  18100 S   0.1   0.3  18:03.14 ros2
    114 root      20   0       0      0      0 S   0.1   0.0  12:21.03 kswapd0
   4916 robot     20   0 2992684  31872  13656 S   0.1   0.2   6:01.08 foxglove_bridge
3634038 root      20   0       0      0      0 I   0.1   0.0   0:06.90 kworker/u16:1-events_unbound
3692395 root      20   0       0      0      0 I   0.1   0.0   0:02.93 kworker/0:2-events
    148 root     -51   0       0      0      0 S   0.0   0.0   5:56.14 irq/127-host_sy
    156 root     -51   0       0      0      0 S   0.0   0.0   4:57.94 irq/135-host_st
    275 root      19  -1   86324  46832  43476 S   0.0   0.3   0:44.66 systemd-journal
    672 root      19  -1       0      0      0 S   0.0   0.0   1:27.14 dp_rx_thread_1
    772 root      20   0    2732   1684   1544 S   0.0   0.0   0:17.85 ptp4l
   1330 root      20   0    2144   1368   1272 S   0.0   0.0   8:46.59 ksz9897_diag
   2966 gdm       20   0  313988  10368   7420 S   0.0   0.1   0:01.19 gsd-housekeepin
   4209 root      20   0    9272   5788   4796 S   0.0   0.0   8:00.65 openvpn
   4668 robot     20   0    7656    404    340 S   0.0   0.0   0:07.93 cat
   4753 root      20   0    2656   1492   1360 S   0.0   0.0   4:10.10 tegrastats
   4973 robot     20   0 2102372   1.8g   1.8g S   0.0  12.1   6:15.57 iox-roudi
   5451 root      20   0       0      0      0 S   0.0   0.0   0:58.88 vi-output, ar02
1390732 root      20   0       0      0      0 I   0.0   0.0   0:07.94 kworker/3:0-events
3540179 root      20   0       0      0      0 I   0.0   0.0   0:17.86 kworker/4:1-pm
3693114 root      20   0       0      0      0 I   0.0   0.0   0:02.26 kworker/u16:4-events_unbound
3698743 systemd+  20   0   24700  11704   7500 S   0.0   0.1   0:00.39 systemd-resolve
3711770 robot     20   0   11952   3680   2692 R   0.0   0.0   0:00.16 top
      1 root      20   0  169788  11076   6888 S   0.0   0.1   1:10.58 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.09 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
      9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_kthre
     10 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
     11 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
     12 root      20   0       0      0      0 S   0.0   0.0   2:11.37 ksoftirqd/0
robot@tegra:~$

perf_report.txt (10.8 KB)

version:

robot@tegra:~$ more /etc/nv_tegra_release
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug  1 19:57:35 UTC 2023

device:
Orin NX

trace.txt (19.2 MB)

ShaneCCC · November 29, 2024, 6:06am

What’s the use case cause it?

wxsrite · November 29, 2024, 6:20am

Hi, ShaneCCC:
The upper layer software is developed based on the GStreamer framework, and the plug-in responsible for outputting images is nvv4l2camerasrc

ShaneCCC · November 30, 2024, 3:47pm

What’s the command line?

wxsrite · December 2, 2024, 6:42am

Hi ShaneCCC:
We are not using the gst-launch-1.0 command, but the GStreamer API.

As follows：

ShaneCCC · December 2, 2024, 8:02am

Suspect the problem could cause by the memory copy.
Are you able to narrow down which element cause the problem.

wxsrite · December 2, 2024, 9:57am

Hi, ShaneCCC:
nvv4l2camerasrc
This is the element.

ShaneCCC · December 2, 2024, 3:41pm

Could you confirm by the comment from below link.

https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SD/Multimedia/AcceleratedGstreamer.html#progressive-capture-using-nvv4l2camerasrc

wxsrite · December 3, 2024, 12:44pm

Hi, ShaneCCC:
I will use the method you provided to verify it. It is not an easy problem to reproduce, and it may not be reproduced in a short time.

Topic		Replies	Views
V4l2 high cpu usage Jetson Xavier NX camera	19	1459	February 21, 2023
Gstreamer / DeepStream nvcamerasrc has high CPU usage (around 60 %) Jetson TX1	6	558	October 18, 2021
When I tested the camera, there was an exception Jetson Orin NX camera	2	24	August 27, 2024
Jetson xavier nx camera IMX477 Jetson Xavier NX camera	13	1570	December 28, 2022
AGX Orin 100% CPU usage when multiple video devices are enabled at the same time, one of the camera devices stops streaming data unexpectedly Jetson AGX Orin camera	30	1741	October 3, 2023
Orin RGB888 Stream Jetson Orin NX camera , device-tree	8	48	October 30, 2024
R35.1 No camera image with imx390 camera Jetson AGX Orin camera , sony-imx390	4	2563	September 28, 2022
IMX219-83 camera stopped working on Jetson Orin Nano Jetson Orin Nano camera	22	67	December 9, 2024
How to investigate csi stream flow Jetson Orin NX camera , kernel	10	494	October 23, 2023
Camera fail on agx Orin 32GB JatPack 5.1.2 Jetson AGX Orin camera , board-design	10	407	January 30, 2024

Why is the CPU usage so high?

Related topics