Hi,
One of my Tx2i unit does not boot at all. Started seeing issue with this unit couple weeks back. I was able to boot the unit in force recovery mode and flash the image successfully that time. But after flashing it won’t boot. The console logs showed some crash -
[ 4.092973] Wake77 for irq=199
[ 4.092974] Wake78 for irq=199
[ 4.092975] Wake79 for irq=199
[ 4.092976] Wake80 for irq=199
[ 4.092976] Wake81 for irq=199
[ 4.092977] Wake82 for irq=199
[ 4.093284] xhci-tegra 3530000.xhci: UTMI port 0 has OTG_CAP
[ 4.093286] xhci-tegra 3530000.xhci: No USB3 port has OTG_CAP
[ 4.114906] spi-tegra114 3210000.spi: Static pin configuration used
[ 4.115060] xhci-tegra 3530000.xhci: Direct firmware load for tegra18x_xusb_firmware failed with error -2
[ 4.115062] xhci-tegra 3530000.xhci: Falling back to user helper
[ 4.115525] spi-tegra114 c260000.spi: Static pin configuration used
[ 4.116200] spi-tegra114 3240000.spi: Static pin configuration used
[ 4.116865] tegra-xotg xotg: OTG rev:0200, ADP:0, SRP:1, HNP:1, RSP:0
[ 4.116900] tegra-xotg xotg: update_id_state: ID floating
[ 4.116904] tegra-xotg xotg: update_vbus_state: VBUS not detected
[ 4.116919] tegra-xotg xotg: Nvidia XUSB OTG Controller
[ 4.118636] tegra-xotg xotg: otg: gadget gadget registered
[ 4.118638] tegra-xotg xotg: set gadget otg_caps from OTG controller
[ 4.118640] tegra-xotg xotg: otg: host not registered yet
[ 4.118642] tegra-xotg xotg: otg: start OTG finite state machine
[ 4.118696] tegra-xudc-new 3550000.xudc: entering ELPG
[ 4.119242] tegra-xudc-new 3550000.xudc: entering ELPG done
[ 4.120214] input: gpio-keys as /devices/gpio-keys/input/input3
[ 4.158353] tegra_rtc c2a0000.rtc: setting system clock to 2000-01-01 01:00:35 UTC (946688435)
[ 4.180545] bpmp: mounted debugfs mirror
[ 4.181089] [dram-timers] DRAM derating cdev registered.
[ 4.185718] spmic-ldo0: disabling
[ 4.185874] spmic-ldo1: disabling
[ 4.186005] en-vdd-sd: disabling
[ 4.186007] en-vdd-cam: disabling
[ 4.186009] vdd-usb0-5v: disabling
[ 4.186011] vdd-usb1-5v: disabling
[ 4.186014] en-vdd-disp-3v3: disabling
[ 4.186015] en-mdm-pwr-3v7: disabling
[ 4.186017] en-vdd-disp-1v8: disabling
[ 4.186018] en-vdd-cam-hv-2v8: disabling
[ 4.186020] en-vdd-cam-1v2: disabling
[ 4.186022] vdd-fan: disabling
[ 4.186023] vdd-3v3: disabling
[ 4.186026] en-vdd-vcm-2v8: disabling
[ 4.186028] vdd-usb2-5v: disabling
[ 4.186029] vdd-sys-bl: disabling
[ 4.186031] en-vdd-sys: disabling
[ 4.191564] ALSA device list:
[ 4.191566] #0: tegra-hda at 0x3518000 irq 400
[ 4.191567] #1: tegra-snd-t186ref-mobile-rt565x
[ 4.192721] tegra-vi4 15700000.vi: initialized
[ 4.193847] tegra-vi4 15700000.vi: subdev 150c0000.nvcsi-2 bound
[ 4.193849] tegra-vi4 15700000.vi: subdev ov5693 2-0036 bound
[ 6.361569] tegradc 15210000.nvdisplay: hdmi: plugged
[ 6.406983] EXT4-fs (mmcblk0p1): warning: mounting fs with errors, running e2fsck is recommended
[ 6.418878] EXT4-fs (mmcblk0p1): Errors on filesystem, clearing orphan list.
[ 6.418878]
[ 6.430018] EXT4-fs (mmcblk0p1): recovery complete
[ 6.437688] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[ 6.447174] VFS: Mounted root (ext4 filesystem) on device 179:1.
[ 6.455820] devtmpfs: mounted
[ 6.460504] Freeing unused kernel memory: 1152K (ffffffc00112f000 - ffffffc00124f000)
[ 6.469726] Freeing alternatives memory: 96K (ffffffc00124f000 - ffffffc001267000)
[ 6.491928] btb inv war enabled
[ 6.544742] systemd[1]: System time before build time, advancing clock.
[ 6.574879] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +)
[ 6.594795] systemd[1]: Detected architecture arm64.
[ 6.612710] systemd[1]: Set hostname to <tegra-ubuntu>.
[ 6.748545] systemd[1]: Created slice User and Session Slice.
[ 6.757945] systemd[1]: Listening on Journal Socket (/dev/log).
[ 6.767424] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[ 6.777922] systemd[1]: Listening on Syslog Socket.
[ 6.786332] systemd[1]: Listening on Journal Socket.
[ 6.794713] systemd[1]: Listening on udev Kernel Socket.
[ 6.803492] systemd[1]: Reached target Encrypted Volumes.
[ 6.814879] systemd[1]: Reached target Swap.
[ 6.823093] systemd[1]: Listening on Journal Audit Socket.
[ 6.832335] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[ 6.843463] systemd[1]: Reached target Remote File Systems (Pre).
[ 6.853304] systemd[1]: Created slice System Slice.
[ 6.861708] systemd[1]: Reached target Slices.
[ 6.871138] systemd[1]: Starting Journal Service...
[ 6.881315] systemd[1]: Mounting POSIX Message Queue File System...
[ 6.896609] systemd[1]: Starting Load Kernel Modules...
[ 6.907285] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 6.915876] EXT4-fs (mmcblk0p1): re-mounted. Opts: (null)
[ 6.925425] systemd[1]: Created slice system-serial\x2dgetty.slice.
[ 6.939475] systemd[1]: Mounting Debug File System...
[ 6.950183] systemd[1]: Starting Set console keymap...
[ 6.960676] systemd[1]: Started Braille Device Support.
[ 6.971877] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[ 6.985375] systemd[1]: Reached target User and Group Name Lookups.
[ 6.996341] systemd[1]: Reached target Remote File Systems.
[ 7.006839] systemd[1]: Listening on udev Control Socket.
[ 7.022384] systemd[1]: Mounted POSIX Message Queue File System.
[ 7.033152] systemd[1]: Mounted Debug File System.
[ 7.043186] systemd[1]: Started Journal Service.
[ 7.207720] systemd-journald[265]: Received request to flush runtime journal from PID 1
[ 7.345726] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
[ 7.346742] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[ 7.347180] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
[ 7.349405] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[ 7.385932] xhci-tegra 3530000.xhci: cannot find firmware....retry after 1 second
[ 7.789798] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[ 8.214702] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[ 8.389678] xhci-tegra 3530000.xhci: Firmware timestamp: 2017-12-07 10:50:08 UTC, Version: 55.09 release
[ 8.391721] xhci-tegra 3530000.xhci: xHCI Host Controller
[ 8.391737] xhci-tegra 3530000.xhci: new USB bus registered, assigned bus number 1
[ 8.392547] xhci-tegra 3530000.xhci: hcc params 0x0184fd25 hci version 0x100 quirks 0x00010810
[ 8.392573] xhci-tegra 3530000.xhci: irq 59, io mem 0x03530000
[ 8.393056] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 8.393059] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.393061] usb usb1: Product: xHCI Host Controller
[ 8.393063] usb usb1: Manufacturer: Linux 4.4.38 xhci-hcd
[ 8.393065] usb usb1: SerialNumber: 3530000.xhci
[ 8.394327] hub 1-0:1.0: USB hub found
[ 8.394351] hub 1-0:1.0: 4 ports detected
[ 8.414799] xhci-tegra 3530000.xhci: xHCI Host Controller
[ 8.414807] xhci-tegra 3530000.xhci: new USB bus registered, assigned bus number 2
[ 8.414931] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[ 8.414934] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.414936] usb usb2: Product: xHCI Host Controller
[ 8.414938] usb usb2: Manufacturer: Linux 4.4.38 xhci-hcd
[ 8.414940] usb usb2: SerialNumber: 3530000.xhci
[ 8.415601] hub 2-0:1.0: USB hub found
[ 8.415623] hub 2-0:1.0: 3 ports detected
[ 8.416395] tegra-xotg xotg: otg: host 3530000.xhci registered
[ 8.624631] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[ 8.634914] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
[ 8.653407] IPVS: Creating netns size=1424 id=1
[ 8.702650] usb 1-2: new high-speed USB device number 2 using xhci-tegra
[ 8.712830] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 8.721664] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 8.774037] tegradc 15210000.nvdisplay: blank - powerdown
[ 8.830917] usb 1-2: feature bit otg_vbus_off set
[ 8.830921] usb 1-2: New USB device found, idVendor=1a40, idProduct=0101
[ 8.830924] usb 1-2: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 8.830926] usb 1-2: Product: USB 2.0 Hub
[ 8.831675] hub 1-2:1.0: USB hub found
[ 8.831740] hub 1-2:1.0: 4 ports detected
[ 8.831987] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 6
[ 8.842856] PD DISP2 index4 DOWN
[ 8.842955] PD DISP1 index3 DOWN
[ 8.843036] PD DISP0 index2 DOWN
[ 8.896486] tegradc 15210000.nvdisplay: unblank
[ 8.896500] PD DISP0 index2 UP
[ 8.897526] PD DISP1 index3 UP
[ 8.897622] PD DISP2 index4 UP
[ 8.899216] Parent Clock set for DC plld2
[ 8.901833] tegradc 15210000.nvdisplay: hdmi: pclk:148500K, set prod-setting:prod_c_150M
[ 9.048470] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 9.102626] usb 1-2.1: new high-speed USB device number 3 using xhci-tegra
[ 9.191000] usb 1-2.1: New USB device found, idVendor=1a40, idProduct=0101
[ 9.191003] usb 1-2.1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 9.191005] usb 1-2.1: Product: USB 2.0 Hub
[ 9.191799] hub 1-2.1:1.0: USB hub found
[ 9.191853] hub 1-2.1:1.0: 4 ports detected
[ 9.278629] usb 1-2.2: new low-speed USB device number 4 using xhci-tegra
[ 9.385995] usb 1-2.2: New USB device found, idVendor=04f2, idProduct=0939
[ 9.385997] usb 1-2.2: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ 9.385999] usb 1-2.2: Product: USB Optical Mouse
[ 9.386192] usb 1-2.2: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[ 9.388909] input: USB Optical Mouse as /devices/3530000.xhci/usb1/1-2/1-2.2/1-2.2:1.0/0003:04F2:0939.0004
[ 9.389049] hid-generic 0003:04F2:0939.0001: input,hidraw0: USB HID v1.11 Mouse [USB Optical Mouse] on us0
[ 9.450817] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 9.486629] usb 1-2.1.4: new low-speed USB device number 5 using xhci-tegra
[ 9.596089] usb 1-2.1.4: New USB device found, idVendor=046d, idProduct=c31c
[ 9.596093] usb 1-2.1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 9.596096] usb 1-2.1.4: Product: USB Keyboard
[ 9.596099] usb 1-2.1.4: Manufacturer: Logitech
[ 9.596354] usb 1-2.1.4: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[ 9.596373] usb 1-2.1.4: ep 0x82 - rounding interval to 1024 microframes, ep desc says 2040 microframes
[ 9.601833] input: Logitech USB Keyboard as /devices/3530000.xhci/usb1/1-2/1-2.1/1-2.1.4/1-2.1.4:1.0/00035
[ 9.655033] hid-generic 0003:046D:C31C.0002: input,hidraw1: USB HID v1.10 Keyboard [Logitech USB Keyboard0
[ 9.662091] input: Logitech USB Keyboard as /devices/3530000.xhci/usb1/1-2/1-2.1/1-2.1.4/1-2.1.4:1.1/00036
[ 9.714786] hid-generic 0003:046D:C31C.0003: input,hidraw2: USB HID v1.10 Device [Logitech USB Keyboard] 1
[ 9.830850] cfg80211: World regulatory domain updated:
[ 9.830856] cfg80211: DFS Master region: unset
[ 9.830856] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
[ 9.830863] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
[ 9.830867] cfg80211: (2457000 KHz - 2482000 KHz @ 20000 KHz, 92000 KHz AUTO), (N/A, 2000 mBm), (N/A)
[ 9.830870] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
[ 9.830873] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
[ 9.830878] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
[ 9.830881] cfg80211: (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
[ 9.830883] cfg80211: (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
[ 9.830886] cfg80211: (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)
[ 9.852689] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 9.854618] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring
[ 9.854625] tegra-pcie 10003000.pcie-controller: PCIE: no end points detected
[ 9.854944] tegra-pcie 10003000.pcie-controller: PCIE: Disable power rails
[ 10.062987] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 5
[ 10.062990] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_DEC_SSPI_CLOCK request
[ 10.276902] tegradc 15210000.nvdisplay: unblank
[ 10.732125] tegradc 15210000.nvdisplay: blank - powerdown
[ 10.790558] PD DISP2 index4 DOWN
[ 10.790660] PD DISP1 index3 DOWN
[ 10.790731] PD DISP0 index2 DOWN
[ 10.814057] tegradc 15210000.nvdisplay: unblank
[ 10.814070] PD DISP0 index2 UP
[ 10.814937] PD DISP1 index3 UP
[ 10.815009] PD DISP2 index4 UP
[ 10.816468] Parent Clock set for DC plld2
[ 10.819088] tegradc 15210000.nvdisplay: hdmi: pclk:185581K, set prod-setting:prod_c_200M
[ 10.819093] tegradc 15210000.nvdisplay: hdmi: pclk:185581K, set prod-setting:prod_c_300M
[ 10.866864] pps_core: source tegra_hsuart2 got cdev (252:0)
[ 10.866868] pps pps0: new PPS source tegra_hsuart2
[ 10.866880] pps pps0: source "/dev/ttyTHS2" added
[ 10.866997] pps pps0: PPS_GETPARAMS
[ 10.867000] pps pps0: PPS_GETCAP
[ 10.867003] pps pps0: PPS_SETPARAMS
[ 10.867190] pps pps0: PPS_FETCH
[ 10.867197] pps pps0: timeout 3.000000000
[ 10.867835] pps pps0: pending signal caught
[ 10.868178] pps pps0: PPS_FETCH
[ 10.868182] pps pps0: timeout 3.000000000
[ 12.017145] gk20a 17000000.gp10b: railgate is disabled.
[ 12.024005] core_bw_settings_store: failed to reserve 8363500 KB/s
[ 12.074743] tegradc 15210000.nvdisplay: blank - powerdown
[ 12.141018] PD DISP2 index4 DOWN
[ 12.141133] PD DISP1 index3 DOWN
[ 12.141227] PD DISP0 index2 DOWN
[ 12.157577] tegradc 15210000.nvdisplay: unblank
[ 12.157592] PD DISP0 index2 UP
[ 12.158645] PD DISP1 index3 UP
[ 12.159234] PD DISP2 index4 UP
[ 12.161004] Parent Clock set for DC plld2
[ 12.163792] tegradc 15210000.nvdisplay: hdmi: pclk:148500K, set prod-setting:prod_c_150M
[ 13.204772] tegradc 15210000.nvdisplay: unblank
[ 13.248775] CPU1: shutdown
[ 13.251597] psci: CPU1 killed.
[ 13.274626] CPU2: shutdown
[ 13.277358] psci: CPU2 killed.
Ubuntu 16.04.5 LTS tegra-ubuntu ttyS0
tegra-ubuntu login: nvidia (automatic login)
Last login: Fri Nov 2 11:20:32 PDT 2018 on ttyS0
[ 13.866768] pps pps0: PPS_FETCH
[ 13.869925] pps pps0: timeout 3.000000000
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.38 aarch64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
nvidia@tegra-ubuntu:~$ [ 14.365242] fuse init (API version 7.23)
[ 16.846425] IPVS: Creating netns size=1424 id=2
[ 16.870732] pps pps0: PPS_FETCH
[ 16.873876] pps pps0: timeout 3.000000000
[ 19.874700] pps pps0: PPS_FETCH
[ 19.877845] pps pps0: timeout 3.000000000
[ 22.878673] pps pps0: PPS_FETCH
[ 22.881810] pps pps0: timeout 3.000000000
[ 25.882673] pps pps0: PPS_FETCH
[ 25.885812] pps pps0: timeout 3.000000000
[ 28.886675] pps pps0: PPS_FETCH
[ 28.889815] pps pps0: timeout 3.000000000
[ 31.890670] pps pps0: PPS_FETCH
[ 31.893807] pps pps0: timeout 3.000000000
[ 38.014611] INFO: rcu_preempt self-detected stall on CPU[ 38.018604] INFO: rcu_preempt detected stalls :
[ 38.018609] 0-...: (5251 ticks this GP) idle=511/140000000000002/0 softirq=2612/2612 fqs=0
[ 38.018610] (detected by 4, t=5252 jiffies, g=450, c=449, q=1101)
[ 38.018614] Task dump for CPU 0:
[ 38.018616] Xorg R running task 0 1078 1068 0x00000003
[ 38.018619] Call trace:
[ 38.018628] [<ffffffc000085eb0>] __switch_to+0x88/0xa0
[ 38.018634] [<ffffffc001435700>] irq_stat+0x0/0x2000
[ 38.018637] rcu_preempt kthread starved for 5252 jiffies! g450 c449 f0x0 s3 ->state=0x1
[ 38.070745]
[ 38.072415] 0-...: (5251 ticks this GP) idle=511/140000000000002/0 softirq=2612/2612 fqs=14
[ 38.080919] (t=5267 jiffies g=450 c=449 q=1101)
[ 38.085620] Task dump for CPU 0:
[ 38.088838] Xorg R running task 0 1078 1068 0x00000003
[ 38.093195] Call trace:
[ 38.093199] [<ffffffc0000893a0>] dump_backtrace+0x0/0xe8
[ 38.093203] [<ffffffc00008949c>] show_stack+0x14/0x20
[ 38.093207] [<ffffffc0000cf8bc>] sched_show_task+0xa4/0x108
[ 38.093210] [<ffffffc0000d1d60>] dump_cpu_task+0x40/0x50
[ 38.093213] [<ffffffc0000ffeb4>] rcu_dump_cpu_stacks+0x9c/0xe8
[ 38.093216] [<ffffffc000103fd0>] rcu_check_callbacks+0x620/0xa18
[ 38.093219] [<ffffffc000108bfc>] update_process_times+0x3c/0x70
[ 38.093222] [<ffffffc0001184b4>] tick_sched_handle.isra.6+0x24/0x78
[ 38.093224] [<ffffffc00011854c>] tick_sched_timer+0x44/0x90
[ 38.093226] [<ffffffc000109564>] __hrtimer_run_queues+0x13c/0x370
[ 38.093229] [<ffffffc000109e38>] hrtimer_interrupt+0xa0/0x1d8
[ 38.093234] [<ffffffc000948c04>] tegra186_timer_isr+0x24/0x30
[ 38.093236] [<ffffffc0000f6154>] handle_irq_event_percpu+0x6c/0x2a0
[ 38.093238] [<ffffffc0000f63d0>] handle_irq_event+0x48/0x78
[ 38.093240] [<ffffffc0000f99b8>] handle_fasteoi_irq+0xb8/0x1b0
[ 38.093242] [<ffffffc0000f578c>] generic_handle_irq+0x24/0x38
[ 38.093243] [<ffffffc0000f5a84>] __handle_domain_irq+0x5c/0xb8
[ 38.093245] [<ffffffc000081774>] gic_handle_irq+0x64/0xc0
[ 38.093247] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 38.093250] [<ffffffc0000a8870>] irq_exit+0x88/0xe0
[ 38.093252] [<ffffffc0000f5a88>] __handle_domain_irq+0x60/0xb8
[ 38.093253] [<ffffffc000081774>] gic_handle_irq+0x64/0xc0
[ 38.093254] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 38.093257] [<ffffffc0000e8230>] __wake_up_sync_key+0x58/0x78
[ 38.093261] [<ffffffc0009d90a0>] sock_def_readable+0x40/0x78
[ 38.093266] [<ffffffc000ac6e60>] unix_stream_sendmsg+0x158/0x330
[ 38.093268] [<ffffffc0009d5550>] sock_sendmsg+0x50/0x60
[ 38.093270] [<ffffffc0009d55c0>] sock_write_iter+0x60/0xb8
[ 38.093274] [<ffffffc0001db94c>] do_iter_readv_writev+0x4c/0x70
[ 38.093276] [<ffffffc0001dc0d4>] do_readv_writev+0x174/0x210
[ 38.093278] [<ffffffc0001dc1e4>] vfs_writev+0x2c/0x48
[ 38.093280] [<ffffffc0001dcdac>] SyS_writev+0x44/0xc8
[ 38.093282] [<ffffffc000084ff0>] el0_svc_naked+0x24/0x28
[ 38.198606] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 38.198610] 0-...: (1 GPs behind) idle=511/140000000000002/0 softirq=2180/2612 fqs=5240
[ 38.198613] (detected by 4, t=5252 jiffies, g=-191, c=-192, q=6)
[ 38.198614] Task dump for CPU 0:
[ 38.198617] Xorg R running task 0 1078 1068 0x00000003
[ 38.198618] Call trace:
[ 38.198623] [<ffffffc000085eb0>] __switch_to+0x88/0xa0
[ 38.198627] [<ffffffc001435700>] irq_stat+0x0/0x2000
[ 44.894756] pps pps0: PPS_FETCH
[ 44.897898] pps pps0: timeout 3.000000000
[ 47.898686] pps pps0: PPS_FETCH
[ 47.901825] pps pps0: timeout 3.000000000
[ 50.902659] pps pps0: PPS_FETCH
[ 50.905796] pps pps0: timeout 3.000000000
[ 53.906653] pps pps0: PPS_FETCH
[ 53.909790] pps pps0: timeout 3.000000000
[ 56.910651] pps pps0: PPS_FETCH
[ 56.913788] pps pps0: timeout 3.000000000
[ 59.914653] pps pps0: PPS_FETCH
[ 59.917789] pps pps0: timeout 3.000000000
[ 62.918656] pps pps0: PPS_FETCH
[ 62.921790] pps pps0: timeout 3.000000000
[ 64.050606] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [Xorg:1078]
[ 64.057724] Modules linked in: fuse pci_tegra bluedroid_pm
[ 64.063233]
[ 64.064721] CPU: 0 PID: 1078 Comm: Xorg Not tainted 4.4.38 #8
[ 64.070453] Hardware name: storm (DT)
[ 64.074106] task: ffffffc1a3f90c80 ti: ffffffc199d28000 task.ti: ffffffc199d28000
[ 64.081574] PC is at __do_softirq+0x98/0x358
[ 64.085834] LR is at irq_exit+0x88/0xe0
[ 64.089659] pc : [<ffffffc0000a8330>] lr : [<ffffffc0000a8870>] pstate: 40000045
[ 64.097040] sp : ffffffc199d2b8f0
[ 64.097047] x29: ffffffc199d2b8f0 x28: ffffffc199d28000
[ 64.097050] x27: ffffffc19533f500 x26: 0000000000000020
[ 64.097052] x25: 0000000000000282 x24: ffffff8000005000
[ 64.097054] x23: ffffffc1adc00800 x22: 0000000000000000
[ 64.097055] x21: 0000000000000000 x20: 0000000000000006
[ 64.097057] x19: ffffffc00124a000 x18: 0000000000000000
[ 64.097059] x17: 0000000000000001 x16: 000000000051a11f
[ 64.097061] x15: 00000000064f6b97 x14: 0000000000000000
[ 64.097063] x13: 00000000004f974a x12: ffffffc000b9d598
[ 64.097064] x11: 0000000000004c4a x10: ffffffc000b9d573
[ 64.097066] x9 : 0000000000000004 x8 : 0000000000000005
[ 64.097068] x7 : 7fffffffffffffff x6 : 00000000ffffffc0
[ 64.097069] x5 : 00000001b4d53000 x4 : ffffffc00124af80
[ 64.097071] x3 : 0000000000000101 x2 : ffffffc000b9cb28
[ 64.097073] x1 : ffffffc001435700 x0 : 0000000000000000
[ 64.097073]
[ 65.922653] pps pps0: PPS_FETCH
[ 65.925787] pps pps0: timeout 3.000000000
[ 68.926673] pps pps0: PPS_FETCH
[ 68.929810] pps pps0: timeout 3.000000000
[ 71.930654] pps pps0: PPS_FETCH
[ 71.933789] pps pps0: timeout 3.000000000
[ 84.934751] pps pps0: PPS_FETCH
[ 84.937891] pps pps0: timeout 3.000000000
[ 87.938683] pps pps0: PPS_FETCH
[ 87.941822] pps pps0: timeout 3.000000000
[ 90.942660] pps pps0: PPS_FETCH
[ 90.945797] pps pps0: timeout 3.000000000
[ 92.050604] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [Xorg:1078]
[ 92.057723] Modules linked in: fuse pci_tegra bluedroid_pm
[ 92.063231]
[ 92.064718] CPU: 0 PID: 1078 Comm: Xorg Tainted: G L 4.4.38 #8
[ 92.071663] Hardware name: storm (DT)
[ 92.075316] task: ffffffc1a3f90c80 ti: ffffffc199d28000 task.ti: ffffffc199d28000
[ 92.082786] PC is at __do_softirq+0x98/0x358
[ 92.087045] LR is at irq_exit+0x88/0xe0
[ 92.090871] pc : [<ffffffc0000a8330>] lr : [<ffffffc0000a8870>] pstate: 40000045
[ 92.098252] sp : ffffffc199d2b8f0
[ 92.098257] x29: ffffffc199d2b8f0 x28: ffffffc199d28000
[ 92.098259] x27: ffffffc19533f500 x26: 0000000000000020
[ 92.098261] x25: 0000000000000282 x24: ffffff8000005000
[ 92.098263] x23: ffffffc1adc00800 x22: 0000000000000000
[ 92.098265] x21: 0000000000000000 x20: 0000000000000006
[ 92.098266] x19: ffffffc00124a000 x18: 0000000000000000
[ 92.098268] x17: 0000000000000001 x16: 000000000051a11f
[ 92.098270] x15: 00000000064f6b97 x14: 0000000000000000
[ 92.098272] x13: 00000000004f974a x12: ffffffc000b9d598
[ 92.098274] x11: 0000000000004c4a x10: ffffffc000b9d573
[ 92.098275] x9 : 0000000000000004 x8 : 0000000000000005
[ 92.098277] x7 : 7fffffffffffffff x6 : 00000000ffffffc0
[ 92.098279] x5 : 00000001b4d53000 x4 : ffffffc00124af80
[ 92.098280] x3 : 0000000000000101 x2 : ffffffc000b9cb28
[ 92.098282] x1 : ffffffc001435700 x0 : 0000000000000000
[ 92.098282]
[ 93.946657] pps pps0: PPS_FETCH
[ 93.949794] pps pps0: timeout 3.000000000
[ 96.950699] pps pps0: PPS_FETCH
[ 96.953841] pps pps0: timeout 3.000000000
[ 98.026608] mmc0: Timeout waiting for hardware interrupt.
[ 98.031997] sdhci: =========== REGISTER DUMP (mmc0)===========
[ 98.037819] sdhci: Sys addr: 0x00000008 | Version: 0x00000404
[ 98.043639] sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[ 98.049459] sdhci: Argument: 0x00811111 | Trn mode: 0x0000003b
[ 98.055277] sdhci: Present: 0x01fb00f0 | Host ctl: 0x00000035
[ 98.061097] sdhci: Power: 0x00000001 | Blk gap: 0x00000000
[ 98.066917] sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 98.072735] sdhci: Timeout: 0x0000000e | Int stat: 0x00000003
[ 98.078554] sdhci: Int enab: 0x02ff000b | Sig enab: 0x02fc000b
[ 98.084374] sdhci: AC12 err: 0x00000000 | Slot int: 0x00000000
[ 98.090193] sdhci: Caps: 0x3f6cd08c | Caps_1: 0x18006f73
[ 98.096012] sdhci: Cmd: 0x0000123a | Max curr: 0x00000000
[ 98.101830] sdhci: Host ctl2: 0x0000300d
[ 98.105743] sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x0000000080000010
[ 98.112253] sdhci: ===========================================
[ 98.118110] ------------[ cut here ]------------
[ 98.122715] WARNING: at ffffffc00084c038 [verbose debug info unavailable]
[ 98.129486] Modules linked in: fuse pci_tegra bluedroid_pm
[ 98.134994]
[ 98.136483] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 4.4.38 #8
[ 98.143603] Hardware name: storm (DT)
[ 98.147255] task: ffffffc1ade82580 ti: ffffffc1ade94000 task.ti: ffffffc1ade94000
[ 98.154727] PC is at sdhci_send_command+0x368/0x530
[ 98.159593] LR is at sdhci_finish_data+0xc8/0x2f8
[ 98.164285] pc : [<ffffffc00084c038>] lr : [<ffffffc00084cc00>] pstate: 400000c5
[ 98.171663] sp : ffffffc1ade97b30
[ 98.174967] x29: ffffffc1ade97b30 x28: ffffffc1ade94000
[ 98.180283] x27: ffffffc1b5fcdc38 x26: dead000000000200
[ 98.185597] x25: ffffffc0013faad0 x24: ffffffc1adda0680
[ 98.190911] x23: ffffffc070292c68 x22: 0000000000000101
[ 98.196225] x21: 0000000000000001 x20: ffffffc070292ca8
[ 98.201539] x19: ffffffc1adda0680 x18: 0000000000000000
[ 98.206854] x17: 0000000000000000 x16: 000000000000030f
[ 98.212168] x15: 0000000000000010 x14: 3d3d3d3d3d3d3d3d
[ 98.217480] x13: 3d3d3d3d3d3d3d3d x12: 0000000000000000
[ 98.222794] x11: 0000000000000001 x10: ffffff8001000064
[ 98.228107] x9 : ffffff8001000000 x8 : 0000000000000000
[ 98.233419] x7 : 0000000000010000 x6 : 00000000ffffffc7
[ 98.238733] x5 : 0000000000000002 x4 : 00000000ffff007f
[ 98.244046] x3 : ffffffc001341960 x2 : 0000000000000000
[ 98.249359] x1 : ffffffc070292c68 x0 : ffffffc070292c28
[ 98.254672]
[ 98.256520] ---[ end trace 3b9a3cde6e2f42a9 ]---
[ 98.261127] Call trace:
[ 98.263569] [<ffffffc00084c038>] sdhci_send_command+0x368/0x530
[ 98.269475] [<ffffffc00084cc00>] sdhci_finish_data+0xc8/0x2f8
[ 98.275207] [<ffffffc00084d060>] sdhci_timeout_timer+0x80/0xd0
[ 98.281031] [<ffffffc0001074dc>] call_timer_fn+0x54/0x1d8
[ 98.286416] [<ffffffc00010788c>] run_timer_softirq+0x214/0x298
[ 98.292237] [<ffffffc0000a83d0>] __do_softirq+0x138/0x358
[ 98.297622] [<ffffffc0000a8870>] irq_exit+0x88/0xe0
[ 98.302489] [<ffffffc0000f5a88>] __handle_domain_irq+0x60/0xb8
[ 98.308308] [<ffffffc000081774>] gic_handle_irq+0x64/0xc0
[ 98.313695] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 98.318477] [<ffffffc00082bf10>] cpuidle_enter+0x18/0x20
[ 98.323776] [<ffffffc0000e8714>] call_cpuidle+0x24/0x50
[ 98.328989] [<ffffffc0000e89b0>] cpu_startup_entry+0x270/0x340
[ 98.334809] [<ffffffc00008e214>] secondary_start_kernel+0x12c/0x168
[ 98.341061] [<0000000080081adc>] 0x80081adc
[ 99.954687] pps pps0: PPS_FETCH
[ 99.957828] pps pps0: timeout 3.000000000
[ 101.030607] INFO: rcu_preempt self-detected stall on CPU
[ 101.035918] 0-...: (20965 ticks this GP) idle=511/140000000000002/0 softirq=2612/2612 fqs=15648
[ 101.038605] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 101.038609] 0-...: (20965 ticks this GP) idle=511/140000000000002/0 softirq=2612/2612 fqs=15648
[ 101.038612] (detected by 4, t=21007 jiffies, g=450, c=449, q=2445)
[ 101.038614] Task dump for CPU 0:
[ 101.038618] Xorg R running task 0 1078 1068 0x00000003
[ 101.038619] Call trace:
[ 101.038625] [<ffffffc000085eb0>] __switch_to+0x88/0xa0
[ 101.038631] [<ffffffc001435700>] irq_stat+0x0/0x2000
[ 101.088267] (t=21019 jiffies g=450 c=449 q=2445)
[ 101.093054] Task dump for CPU 0:
[ 101.096273] Xorg R running task 0 1078 1068 0x00000003
[ 101.103323] Call trace:
[ 101.105764] [<ffffffc0000893a0>] dump_backtrace+0x0/0xe8
[ 101.111064] [<ffffffc00008949c>] show_stack+0x14/0x20
[ 101.116106] [<ffffffc0000cf8bc>] sched_show_task+0xa4/0x108
[ 101.121665] [<ffffffc0000d1d60>] dump_cpu_task+0x40/0x50
[ 101.126966] [<ffffffc0000ffeb4>] rcu_dump_cpu_stacks+0x9c/0xe8
[ 101.132785] [<ffffffc000103fd0>] rcu_check_callbacks+0x620/0xa18
[ 101.138777] [<ffffffc000108bfc>] update_process_times+0x3c/0x70
[ 101.144684] [<ffffffc0001184b4>] tick_sched_handle.isra.6+0x24/0x78
[ 101.150936] [<ffffffc00011854c>] tick_sched_timer+0x44/0x90
[ 101.156496] [<ffffffc000109564>] __hrtimer_run_queues+0x13c/0x370
[ 101.162575] [<ffffffc000109e38>] hrtimer_interrupt+0xa0/0x1d8
[ 101.168312] [<ffffffc000948c04>] tegra186_timer_isr+0x24/0x30
[ 101.174045] [<ffffffc0000f6154>] handle_irq_event_percpu+0x6c/0x2a0
[ 101.180298] [<ffffffc0000f63d0>] handle_irq_event+0x48/0x78
[ 101.185857] [<ffffffc0000f99b8>] handle_fasteoi_irq+0xb8/0x1b0
[ 101.191676] [<ffffffc0000f578c>] generic_handle_irq+0x24/0x38
[ 101.197408] [<ffffffc0000f5a84>] __handle_domain_irq+0x5c/0xb8
[ 101.203226] [<ffffffc000081774>] gic_handle_irq+0x64/0xc0
[ 101.208612] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 101.213392] [<ffffffc0000a8870>] irq_exit+0x88/0xe0
[ 101.218256] [<ffffffc0000f5a88>] __handle_domain_irq+0x60/0xb8
[ 101.218605] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 101.218609] 0-...: (1 GPs behind) idle=511/140000000000002/0 softirq=2180/2612 fqs=20890
[ 101.218612] (detected by 3, t=21007 jiffies, g=-191, c=-192, q=6)
[ 101.218613] Task dump for CPU 0:
[ 101.218616] Xorg R running task 0 1078 1068 0x00000003
[ 101.218617] Call trace:
[ 101.218620] [<ffffffc000085eb0>] __switch_to+0x88/0xa0
[ 101.218623] [<ffffffc001435700>] irq_stat+0x0/0x2000
[ 101.266707] [<ffffffc000081774>] gic_handle_irq+0x64/0xc0
[ 101.272091] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 101.276870] [<ffffffc0000e8230>] __wake_up_sync_key+0x58/0x78
[ 101.282603] [<ffffffc0009d90a0>] sock_def_readable+0x40/0x78
[ 101.288253] [<ffffffc000ac6e60>] unix_stream_sendmsg+0x158/0x330
[ 101.294247] [<ffffffc0009d5550>] sock_sendmsg+0x50/0x60
[ 101.299459] [<ffffffc0009d55c0>] sock_write_iter+0x60/0xb8
[ 101.304933] [<ffffffc0001db94c>] do_iter_readv_writev+0x4c/0x70
[ 101.310837] [<ffffffc0001dc0d4>] do_readv_writev+0x174/0x210
[ 101.316482] [<ffffffc0001dc1e4>] vfs_writev+0x2c/0x48
[ 101.321521] [<ffffffc0001dcdac>] SyS_writev+0x44/0xc8
[ 101.326560] [<ffffffc000084ff0>] el0_svc_naked+0x24/0x28
[ 102.958672] pps pps0: PPS_FETCH
[ 102.961811] pps pps0: timeout 3.000000000
[ 105.962655] pps pps0: PPS_FETCH
[ 105.965790] pps pps0: timeout 3.000000000
[ 108.042606] mmc0: Timeout waiting for hardware interrupt.
[ 108.047994] sdhci: =========== REGISTER DUMP (mmc0)===========
[ 108.053816] sdhci: Sys addr: 0x00000008 | Version: 0x00000404
[ 108.059636] sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[ 108.065455] sdhci: Argument: 0x00000000 | Trn mode: 0x00000033
[ 108.071276] sdhci: Present: 0x01fb00f1 | Host ctl: 0x00000035
[ 108.077095] sdhci: Power: 0x00000001 | Blk gap: 0x00000000
[ 108.082914] sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 108.088732] sdhci: Timeout: 0x0000000e | Int stat: 0x00018001
[ 108.094553] sdhci: Int enab: 0x02ff000b | Sig enab: 0x02fc000b
[ 108.094558] sdhci: AC12 err: 0x00000+-----------------------------+
[ 108.094560] sdhci: Caps: 0x3f6cd| |
[ 108.094562] sdhci: Cmd: 0x00000| Cannot open /dev/ttyUSB0! |
[ 108.094564] sdhci: Host ctl2: 0x0000| |
[ 108.094567] sdhci: ADMA Err: 0x00000+-----------------------------+010
[ 108.094568] sdhci: ===========================================
[ 108.966655] pps pps0: PPS_FETCH
[ 108.969791] pps pps0: timeout 3.000000000
[ 111.970651] pps pps0: PPS_FETCH
[ 111.973787] pps pps0: timeout 3.000000000
[ 118.122606] mmc0: Timeout waiting for hardware interrupt.
[ 118.127993] sdhci: =========== REGISTER DUMP (mmc0)===========
[ 118.133814] sdhci: Sys addr: 0x00000008 | Version: 0x00000404
[ 118.139635] sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[ 118.145455] sdhci: Argument: 0x00010000 | Trn mode: 0x00000033
[ 118.151274] sdhci: Present: 0x01fb00f0 | Host ctl: 0x00000035
[ 118.157094] sdhci: Power: 0x00000001 | Blk gap: 0x00000000
[ 118.162913] sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 118.168733] sdhci: Timeout: 0x0000000e | Int stat: 0x00000001
[ 118.174552] sdhci: Int enab: 0x02ff000b | Sig enab: 0x02fc000b
[ 118.180371] sdhci: AC12 err: 0x00000000 | Slot int: 0x00000000
[ 118.186191] sdhci: Caps: 0x3f6cd08c | Caps_1: 0x18006f73
[ 118.192009] sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000
[ 118.197829] sdhci: Host ctl2: 0x0000300d
[ 118.201742] sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x0000000080000010
[ 118.208253] sdhci: ===========================================
[ 118.214166] mmcblk0: error -110 sending status command, retrying
Just saved the logs but didn’t debug further at that point. Started looking at the unit now to debug the problem but can’t boot the unit in normal or recovery mode and I can’t connect to console. Here are the messages I see in the dmesg for the UART cable on my PC -
$ dmesg | grep tty
---
---
[109894.069995] ftdi_sio ttyUSB0: error from flowcontrol urb
[109894.070241] ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
[109894.561836] usb 1-3: FTDI USB Serial Device converter now attached to ttyUSB0
[109909.657311] [<ffffffff81506cf1>] tty_ioctl+0x5f1/0xca0
[109909.657315] [<ffffffff81503db1>] ? tty_write_unlock+0x31/0x40
[109909.657317] [<ffffffff8150e046>] ? tty_ldisc_deref+0x16/0x20
[109917.548640] ftdi_sio ttyUSB0: failed to get modem status: -110
[109917.550486] ftdi_sio ttyUSB0: error from flowcontrol urb
[109917.551029] ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
Not sure what is going on and how to debug the issue. Any suggestion ?
Thanks
