high cpu0 usage with high resolution usb 3 camera

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

I am seeing 100% usage on CPU 0 when I set my USB 3 camera to “streaming” in my TX2. I am not doing anything with the data, only turn the stream of the camera on, sleep for a few seconds and turn it back off.

That is a problem, because the framerate of my camera is quite low.

The output of top shows:

2637 root       0 -20       0      0      0 R 100.0  0.0   0:38.38 kworker/0:2H

Looking at the interrupts:

CPU0       CPU3       CPU4       CPU5       
  3:          0          0          0          0     GICv2  30 Edge      arch_timer
  6:      49429          0          0          0     GICv2  32 Level     tegra186_timer0
  7:          0          0          0          0     GICv2  33 Level     tegra186_timer1
  8:          0          0          0          0     GICv2  34 Level     tegra186_timer2
  9:          0      84772          0          0     GICv2  35 Level     tegra186_timer3
 10:          0          0      76227          0     GICv2  36 Level     tegra186_timer4
 11:          0          0          0      86816     GICv2  37 Level     tegra186_timer5
 12:      23296          0          0          0     GICv2 208 Level     hsp
 13:          0          0          0          0     GICv2 202 Level     arm-smmu global fault
 14:          0          0          0          0     GICv2 203 Level     arm-smmu global fault
 22:      21121          0          0          0     GICv2  97 Level     3460000.sdhci
 23:        749          0          0          0     GICv2  96 Level     3440000.sdhci
 24:          0          0          0          0     GICv2  94 Level     3400000.sdhci
 25:          0          0          0          0     GICv2 229 Level     3507000.ahci-sata
 26:        245          0          0          0     GICv2  57 Level     3160000.i2c
 27:          3          0          0          0     GICv2  58 Level     c240000.i2c
 28:          3          0          0          0     GICv2  59 Level     3180000.i2c
 29:         72          0          0          0     GICv2  60 Level     3190000.i2c
 30:          0          0          0          0     GICv2  62 Level     31b0000.i2c
 31:          0          0          0          0     GICv2  63 Level     31c0000.i2c
 32:        103          0          0          0     GICv2  64 Level     c250000.i2c
 33:          0          0          0          0     GICv2  65 Level     31e0000.i2c
 34:          0          0          0          0     GICv2  68 Level     3210000.spi
 35:          0          0          0          0     GICv2  69 Level     c260000.spi
 36:          0          0          0          0     GICv2  71 Level     3240000.spi
 37:       1325          0          0          0     GICv2 144 Level     serial
 42:        883          0          0          0     GICv2 226 Level     ether_qos.common_irq
 44:        239          0          0          0     GICv2 222 Level     2490000.ether_qos.rx0
 45:         65          0          0          0     GICv2 218 Level     2490000.ether_qos.tx0
 52:          0          0          0          0     GICv2  48 Level     b000000.rtcpu
 53:        176          0          0          0     GICv2 242 Level     d230000.actmon
 54:          0          0          0          0        PM  42 Level     tegra_rtc
 55:          0          0          0          0     GICv2 255 Level     mc_status
 57:          1          0          0          0     GICv2 196 Level     3538000.mailbox
 59:      33599          0          0          0        PM 195 Level     xhci-hcd:usb1
 60:          0          0          0          0        PM 199 Level     3530000.xhci, xotg
 61:          0          0          0          0     GICv2 198 Level     3550000.xudc
 62:      88951          0          0          0     GICv2 297 Level     host_syncpt
 63:          0          0          0          0     GICv2 295 Level     host_status
 64:          0          0          0          0     GICv2 151 Level     150c0000.nvcsi
 65:          0          0          0          0     GICv2 233 Level     15700000.vi
 68:          0          0          0          0     GICv2 237 Level     tegra-isp-isr
 69:      34643          0          0          0     GICv2 186 Level     15210000.nvdisplay
 73:         18          0          0          0     GICv2 102 Level     gk20a_stall
 74:          0          0          0          0     GICv2 103 Level     gk20a_nonstall
 76:          0          0          0          0     GICv2 315 Level     3ad0000.se_elp
 77:         54          0          0          0     GICv2 173 Level     b150000.tegra-hsp
 78:         53          0          0          0     GICv2 174 Level     b150000.tegra-hsp, b150000.tegra-hsp
 81:          0          0          0          0     GICv2 165 Level     c150000.tegra-hsp
 92:          6          0          0          0     GICv2 107 Level     gpcdma.0
 93:          6          0          0          0     GICv2 108 Level     gpcdma.1
 94:          0          0          0          0     GICv2 109 Level     gpcdma.2
 95:          0          0          0          0     GICv2 110 Level     gpcdma.3
 96:          0          0          0          0     GICv2 111 Level     gpcdma.4
 97:          0          0          0          0     GICv2 112 Level     gpcdma.5
 98:          0          0          0          0     GICv2 113 Level     gpcdma.6
 99:          0          0          0          0     GICv2 114 Level     gpcdma.7
100:          0          0          0          0     GICv2 115 Level     gpcdma.8
101:          0          0          0          0     GICv2 116 Level     gpcdma.9
102:          0          0          0          0     GICv2 117 Level     gpcdma.10
103:          0          0          0          0     GICv2 118 Level     gpcdma.11
104:          0          0          0          0     GICv2 119 Level     gpcdma.12
105:          0          0          0          0     GICv2 120 Level     gpcdma.13
106:          0          0          0          0     GICv2 121 Level     gpcdma.14
107:          0          0          0          0     GICv2 122 Level     gpcdma.15
108:          0          0          0          0     GICv2 123 Level     gpcdma.16
109:          0          0          0          0     GICv2 124 Level     gpcdma.17
110:          0          0          0          0     GICv2 125 Level     gpcdma.18
111:          0          0          0          0     GICv2 126 Level     gpcdma.19
112:          0          0          0          0     GICv2 127 Level     gpcdma.20
113:          0          0          0          0     GICv2 128 Level     gpcdma.21
114:          0          0          0          0     GICv2 129 Level     gpcdma.22
115:          0          0          0          0     GICv2 130 Level     gpcdma.23
116:          0          0          0          0     GICv2 131 Level     gpcdma.24
117:          0          0          0          0     GICv2 132 Level     gpcdma.25
118:          0          0          0          0     GICv2 133 Level     gpcdma.26
119:          0          0          0          0     GICv2 134 Level     gpcdma.27
120:          0          0          0          0     GICv2 135 Level     gpcdma.28
121:          0          0          0          0     GICv2 136 Level     gpcdma.29
122:          0          0          0          0     GICv2 137 Level     gpcdma.30
123:          0          0          0          0     GICv2 138 Level     gpcdma.31
232:         13          0          0          0  tegra-gpio 101 Level     phy_interrupt
252:          2          0          0          0  tegra-gpio 121 Edge      15210000.nvdisplay
256:          0          0          0          0  tegra-gpio 125 Edge      3400000.sdhci cd
290:          1          0          0          0  tegra-gpio 159 Edge      external-connection:extcon@1
340:          0          0          0          0  tegra-gpio-aon  16 Level     tmp451
380:          0          0          0          0  tegra-gpio-aon  56 Edge      Power
381:          0          0          0          0  tegra-gpio-aon  57 Edge      Volume Up
382:          0          0          0          0  tegra-gpio-aon  58 Edge      Volume Down
384:          1          0          0          0  tegra-gpio-aon  60 Edge      bluetooth hostwake
390:         54          0          0          0       GIC  32 Level     adma.0
391:         53          0          0          0       GIC  33 Level     adma.1
392:          0          0          0          0       GIC  34 Level     adma.2
393:          0          0          0          0       GIC  35 Level     adma.3
394:          0          0          0          0       GIC  36 Level     adma.4
395:          0          0          0          0       GIC  37 Level     adma.5
396:          0          0          0          0       GIC  38 Level     adma.6
397:          0          0          0          0       GIC  39 Level     adma.7
398:          0          0          0          0       GIC  40 Level     adma.8
399:          0          0          0          0       GIC  41 Level     adma.9
400:        588          0          0          0     GICv2 193 Level     snd_hda_tegra
411:          0          0          0          0       GIC  73 Edge      hwmbox1_send_empty
412:          0          0          0          0       GIC  64 Edge      hwmbox0_recv_full
413:          0          0          0          0       GIC 115 Edge      adsp watchdog
414:          0          0          0          0       GIC  94 Edge      adsp wfi
415:          0          0          0          0       GIC  89 Level     AMC error int
424:         20          0          0          0     GICv2  39 Level     30c0000.watchdog
429:          0          0          0          0        PM 241 Edge      max77620-top
433:          0          0          0          0  max77620-top   3 Edge      max77620-gpio
434:          0          0          0          0  max77620-top   4 Edge      max77686-rtc
438:          0          0          0          0  max77620-top   8 Edge      max77620-thermal
439:          0          0          0          0  max77620-top   9 Edge      max77620-thermal
440:          0          0          0          0  max77620-gpio   0 Edge      external-connection:extcon@1
448:          0          0          0          0  max77686-rtc   1 Edge      rtc-alarm1
IPI0:     60058     104939     133709      91529       Rescheduling interrupts
IPI1:       190         33         88         35       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       Timer broadcast interrupts
IPI4:      1607       1264       1555       1076       IRQ work interrupts
Err:          0

And looking at dmesg, I see a few like this:

[ 1075.350524] INFO: rcu_preempt self-detected stall on CPU
[ 1075.355896] 	0-...: (2530 ticks this GP) idle=62f/140000000000002/0 softirq=14440/14440 fqs=2631 
[ 1075.364762] 	 (t=5250 jiffies g=5332 c=5331 q=4440)
[ 1075.369671] Task dump for CPU 0:
[ 1075.372907] kworker/0:1H    R  running task        0   248      2 0x00000002
[ 1075.380015] Call trace:
[ 1075.382495] [<ffffffc000089388>] dump_backtrace+0x0/0xe8
[ 1075.387820] [<ffffffc000089484>] show_stack+0x14/0x20
[ 1075.392891] [<ffffffc0000cfaf4>] sched_show_task+0xa4/0x108
[ 1075.398471] [<ffffffc0000d1f60>] dump_cpu_task+0x40/0x50
[ 1075.403793] [<ffffffc0001007a8>] rcu_dump_cpu_stacks+0x98/0xe0
[ 1075.409634] [<ffffffc000104838>] rcu_check_callbacks+0x638/0xa28
[ 1075.415651] [<ffffffc000109484>] update_process_times+0x3c/0x70
[ 1075.421578] [<ffffffc000118dfc>] tick_sched_handle.isra.6+0x24/0x78
[ 1075.427850] [<ffffffc000118e94>] tick_sched_timer+0x44/0x90
[ 1075.433434] [<ffffffc000109e0c>] __hrtimer_run_queues+0x13c/0x370
[ 1075.439537] [<ffffffc00010a6f8>] hrtimer_interrupt+0xa0/0x1d8
[ 1075.445297] [<ffffffc000953e94>] tegra186_timer_isr+0x24/0x30
[ 1075.451052] [<ffffffc0000f6a24>] handle_irq_event_percpu+0x6c/0x2a0
[ 1075.457322] [<ffffffc0000f6ca0>] handle_irq_event+0x48/0x78
[ 1075.462901] [<ffffffc0000fa238>] handle_fasteoi_irq+0xb8/0x1b0
[ 1075.468741] [<ffffffc0000f6054>] generic_handle_irq+0x24/0x38
[ 1075.474493] [<ffffffc0000f6354>] __handle_domain_irq+0x5c/0xb8
[ 1075.480332] [<ffffffc0000815cc>] gic_handle_irq+0x64/0xc0
[ 1075.485737] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 1075.490543] [<ffffffc0006bd4fc>] eqos_napi_mq+0xa4/0x2b0
[ 1075.495872] [<ffffffc0009fe9c0>] net_rx_action+0x1d0/0x340
[ 1075.501368] [<ffffffc0000a831c>] __do_softirq+0x124/0x350
[ 1075.506772] [<ffffffc0000a87c8>] irq_exit+0x88/0xe0
[ 1075.511656] [<ffffffc0000f6358>] __handle_domain_irq+0x60/0xb8
[ 1075.517492] [<ffffffc0000815cc>] gic_handle_irq+0x64/0xc0
[ 1075.522897] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 1075.527701] [<ffffffc000b9a9ec>] schedule+0x44/0xb8
[ 1075.532588] [<ffffffc0000bc5d4>] worker_thread+0x1ac/0x498
[ 1075.538082] [<ffffffc0000c22ac>] kthread+0xdc/0xf0
[ 1075.542882] [<ffffffc000084f90>] ret_from_fork+0x10/0x40

Any idea or suggestion will be appreciated.

2

Hi,
Do you run gstreamer pipeline or tegra_multimedia_api?

3

None of that. I have two versions of software in c++ that do the same (open the stream), one uses only v4l2 and the other one uses the api of the camera (they behave the same so I assume the api of the camera is just a wrapper over v4l2).

I’m not even displaying the video or doing anything with the data.

The camera is 4112x3008, and I’m trying to use it at 30fps. The max I can get now due to the high cpu utilization is 10-12 fps.

4

Actually my v4l2 driver is based on the tegra api sample “v4l2” using mmap, with very minor modifications. So we can say that I am using the tegra_multimedia_api.

Any suggestion will be welcomed.

5

Can you please try tegra_multimedia_api\samples\12_camera_v4l2_cuda ?

6

Unfortunately I can’t without modifications, as the camera doesn’t seem to support DMABUFF. Also, the output of my camera is BA81, not supported by that program.

I am using the camera using MMAP. My program is minor modification of the other sample, v4l2cuda.

7

Some additional tests using:

v4l2-ctl --set-fmt-video=width=4112,height=3008,pixelformat="BA81" --device=/dev/video0 --stream-count=3000 --stream-mmap

The exposure is manual, set to the min. Framerate set to max (41fps). Verified that the port is detected as USB 3. Tested with 3 different USB cables with similar results. Tested in Windows with the same cables, getting 41fps.

In the TX2 the fps are around 10fps, and using top at the same time I see different processes using 100% of cpu0, alternating between them.

Some times is kworker/0:0H, sometimes is watchdog 0.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                             
   11 root      rt   0       0      0      0 S  50.0  0.0   0:48.61 watchdog/0                                                          
 3856 root       0 -20       0      0      0 R  50.0  0.0   4:59.02 kworker/0:0H

Not sure what else to test at this point.

8

Hi,
Does your USB3 camera only can output Bayer format BA81? For USB cameras, we suggest the format to be YUV422 such as UYVY, YUYV so that we can put it into DMA buffers.

For Bayer format, valid usecases is to connect the sensor to CSI port directly so that HW ISP engine can be used for processing.

Bayer-format frames through USB interfaces are totally CPU-based stack. Only thing you can try is to run jetson_clocks.sh to let CPUs running at max clocks.

9

DaneLLL,

Thanks for your answer. Running jetson_clocks.sh allows me to run at 30fps.

v4l2-ctl --set-fmt-video=width=4112,height=3008,pixelformat="BA81" --device=/dev/video0 --stream-count=3000 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.39 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.50 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.46 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.50 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.53 fps

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
   57 root     -51   0       0      0      0 S  41.7  0.0   6:08.36 irq/62-hos+

It’s still heavy on the CPU, but at least it works and I can continue the development. I will check with the manufacturer of the camera, as I might have to change to a different one for the final product, that can output YUV as you suggested.

Thanks a lot.