AGX Xavier reboots after 1 core cpu loads 100%

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

When I run cameras stream with argus_camera app or gst-launch-1.0 nvarguscamerasrc. After a few time the first core of CPU loads 100% and jetson reboots. When I stop displaying its work longer, but reboots.
I find same toopic but it didnt help me:

image
Previos topic: How much cameras AXG Xavier can processed? - #26 by kuskov.stanislav

Hardware: AGX Xavier, conecttech gmsl board JCB004, 2 leopard imx390 cameras
Software: Jetpack 4.4, CTI drivers for JCB004

nvpmodel -m 0
nvpmodel -d cool
sudo jetson_clocks

3

I’m try argus_camera --outputsize=640x480 and it works well more 5 hours

4

Thanks for confirm it. Please update it if can run more times.

5

Could you check the CPU usage is still 100% for 640x480 output size?

6

It works near 5 hours and CPU does not load 100%.
Also, I try to run some examples from /usr/src/nvidia/graphics_demos and it’s not load CPU and no reboots

Right now i want to try argus_camera --outputsize=640x480 with multiple session

7

argus_camera --outputsize=640x480 freezes too (multiple session).
I find same pretty same problem vithout solution here: Argus errors with high cpu load and subsequent issues - #24 by SanjayD

Logs same:

Jan 27 11:52:55 jetson nvargus-daemon[6225]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Jan 27 11:52:55 jetson nvargus-daemon[6225]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)

PS. I see what syslog file is large

8

I’m try to start my opencv gstreamer example and all works well vore 4 hours, but when i connect wifi usb adapter it freezes.
Syslog here: (before HDMI reconnection)
err.txt (230.5 KB)

9

I try to restart stream when usb wifi adapter was connected and Jetson reboots. Log here: after Jan 28 02:11:28 jetson kernel: [19699.847822] imx390 32-001c: imx390_set_gain: db: 84
syslog (558.8 KB)

10

It can be interesting too. Syslog before reboot. At now Errors in nvargus_daemon. Here I’m turn off time synchronization (do-timestamp=false)

err_nvargus.txt (97.9 KB)

11

log when i started argus_camera --outputsize=640x480
syslog (539.3 KB)

12

@ShaneCCC Any ideas?

13

Could you try below command before run the camera.

sudo systemctl stop nvphs.service
sudo systemctl list-units --type=service
14 
UNIT                               LOAD   ACTIVE SUB     DESCRIPTION                                                       
accounts-daemon.service            loaded active running Accounts Service                                                  
alsa-restore.service               loaded active exited  Save/Restore Sound Card State                                     
apport.service                     loaded active exited  LSB: automatic crash report generation                            
avahi-daemon.service               loaded active running Avahi mDNS/DNS-SD Stack                                           
bluetooth.service                  loaded active running Bluetooth service                                                 
bolt.service                       loaded active running Thunderbolt system service                                        
colord.service                     loaded active running Manage, Install and Generate Color Profiles                       
console-setup.service              loaded active exited  Set console font and keymap                                       
containerd.service                 loaded active running containerd container runtime                                      
cron.service                       loaded active running Regular background program processing daemon                      
dbus.service                       loaded active running D-Bus System Message Bus                                          
fwupd.service                      loaded active running Firmware update daemon                                            
gdm.service                        loaded active running GNOME Display Manager                                             
grub-common.service                loaded active exited  LSB: Record successful boot for GRUB                              
haveged.service                    loaded active running Entropy daemon using the HAVEGE algorithm                         
kerneloops.service                 loaded active running Tool to automatically collect and submit kernel crash signatures  
keyboard-setup.service             loaded active exited  Set the console keyboard layout                                   
kmod-static-nodes.service          loaded active exited  Create list of required static device nodes for the current kernel
ModemManager.service               loaded active running Modem Manager                                                     
networkd-dispatcher.service        loaded active running Dispatcher daemon for systemd-networkd                            
networking.service                 loaded active exited  Raise network interfaces                                          
NetworkManager.service             loaded active running Network Manager                                                   
nv-l4t-usb-device-mode.service     loaded active exited  Configure USB flashing port for device mode                       
nvargus-daemon.service             loaded active running Argus daemon                                                      
nvgetty.service                    loaded active running UART on ttyTHS0                                                   
nvs-service.service                loaded active running NVS-SERVICE Embedded Sensor HAL Daemon                            
packagekit.service                 loaded active running PackageKit Daemon                                                 
polkit.service                     loaded active running Authorization Manager                                             
resolvconf.service                 loaded active exited  Nameserver information manager                                    
rpcbind.service                    loaded active running RPC bind portmap service                                          
rsyslog.service                    loaded active running System Logging Service                                            
rtkit-daemon.service               loaded active running RealtimeKit Scheduling Policy Service                             
serial-getty@ttyGS0.service        loaded active running Serial Getty on ttyGS0                                            
serial-getty@ttyTCU0.service       loaded active running Serial Getty on ttyTCU0                                           
setvtrgb.service                   loaded active exited  Set console scheme                                                
snapd.seeded.service               loaded active exited  Wait until snapd is fully seeded                                  
speech-dispatcher.service          loaded active exited  LSB: Speech Dispatcher                                            
ssh.service                        loaded active running OpenBSD Secure Shell server                                       
sysstat.service                    loaded active exited  Resets System Activity Data Collector                             
systemd-journal-flush.service      loaded active exited  Flush Journal to Persistent Storage                               
systemd-journald.service           loaded active running Journal Service                                                   
systemd-logind.service             loaded active running Login Service                                                     
systemd-modules-load.service       loaded active exited  Load Kernel Modules                                               
systemd-random-seed.service        loaded active exited  Load/Save Random Seed                                             
systemd-remount-fs.service         loaded active exited  Remount Root and Kernel File Systems                              
systemd-resolved.service           loaded active running Network Name Resolution                                           
systemd-sysctl.service             loaded active exited  Apply Kernel Variables                                            
systemd-timesyncd.service          loaded active running Network Time Synchronization                                      
systemd-tmpfiles-setup-dev.service loaded active exited  Create Static Device Nodes in /dev                                
systemd-tmpfiles-setup.service     loaded active exited  Create Volatile Files and Directories                             
systemd-udev-trigger.service       loaded active exited  udev Coldplug all Devices                                         
systemd-udevd.service              loaded active running udev Kernel Device Manager                                        
systemd-update-utmp.service        loaded active exited  Update UTMP about System Boot/Shutdown                            
systemd-user-sessions.service      loaded active exited  Permit User Sessions                                              
ubuntu-fan.service                 loaded active exited  Ubuntu FAN network setup                                          
udisks2.service                    loaded active running Disk Manager                                                      
upower.service                     loaded active running Daemon for power management                                       
user@1000.service                  loaded active running User Manager for UID 1000                                         
whoopsie.service                   loaded active running crash report submission daemon                                    
wpa_supplicant.service             loaded active running WPA supplicant                                                    

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

60 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
15

@ShaneCCC
I find problem in bluetooth hostwake (cpu irq). How to turn off it forever?
I try disable bluetooth.service but it not help
I try this:

sudo modprobe -r bluedroid_pm

but it works before reboot

16

@WayneWWW
Do you have idea to disable buletooth interrupt for debug.
Or any experience for bluetooth cause CPU loading?

Thanks

17

Hello,

You can directly remove bluedroid_pm.ko file from /lib/modules.

1 Like
18

So, I’m add install bluedroid_pm /bin/false into /etc/modprobe.d/blacklist.conf.
It can stop this module and dependencies. Works well.

Thank You for Your help @ShaneCCC @WayneWWW !

1 Like
19

BTW, could you check if jetpack4.5 also has this issue on your board? This should be resolved on jp4.5.

20

I can not because camera drivers with conecttech gmsl board available for JP 4.3 and 4.4 only

1 Like
21

I add a repo with a description of all fixes for the camera work.
Hope it helps other developers

1 Like