There seems to be something wrong with t18x_a57_enter_state function

Hi everyone:
We are using tx2 jetpack3.2.1 L4T r28.2.1.We have a 3rd party driver tw6869 it can work, but randomly crashed.According to the kernel log:

[  203.958544] Watchdog detected hard LOCKUP on cpu 0
[  203.963208] ------------[ cut here ]------------
[  203.968036] WARNING: at ffffffc00013edf8 [verbose debug info unavailable]
[  203.974833] Modules linked in: tw6869(O) fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd bluedroid_pm
[  203.993395] 
[  203.994914] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G           O    4.4.38 #5
[  204.002058] Hardware name: quill (DT)
[  204.005736] task: ffffffc1ece83e80 ti: ffffffc1ece9c000 task.ti: ffffffc1ece9c000
[  204.013245] PC is at watchdog_timer_fn+0x230/0x33c
[  204.018051] LR is at watchdog_timer_fn+0x230/0x33c
[  204.022856] pc : [<ffffffc00013edf8>] lr : [<ffffffc00013edf8>] pstate: 600001c5
[  204.030256] sp : ffffffc1ece9fae0
[  204.033582] x29: ffffffc1ece9fae0 x28: 0000000000000000 
[  204.038934] x27: ffffffc00126db30 x26: ffffffc1f5ff1278 
[  204.044283] x25: ffffffc00123d2d8 x24: ffffffc1ece9fdc0 
[  204.049631] x23: 0000000000000000 x22: 00000000000000ba 
[  204.054978] x21: ffffffc00126d000 x20: ffffffc00123d000 
[  204.060324] x19: ffffffc00123d260 x18: 0000000000000a03 
[  204.065672] x17: 0000000000ad4878 x16: ffffffc00011a8fc 
[  204.071018] x15: 0000000000000040 x14: 0000000000000000 
[  204.076363] x13: 0000000000000000 x12: 000000000000a800 
[  204.081709] x11: 000000000000a800 x10: 0000000000010002 
[  204.087056] x9 : 000000000000043d x8 : 0000000000000000 
[  204.092402] x7 : ffffffc0012afd00 x6 : ffffffc001285738 
[  204.097748] x5 : 0000000000000000 x4 : 0000000000000000 
[  204.103093] x3 : 0000000000000000 x2 : 0000000000010001 
[  204.108439] x1 : ffffffc1ece9c000 x0 : 0000000000000026 
[  204.113784] 
[  204.116364] ---[ end trace 937c384bd811378a ]---
[  204.120996] Call trace:
[  204.123470] [<ffffffc00013edf8>] watchdog_timer_fn+0x230/0x33c
[  204.129332] [<ffffffc000107d64>] __hrtimer_run_queues+0x140/0x350
[  204.135442] [<ffffffc0001087c4>] hrtimer_interrupt+0x9c/0x1e0
[  204.141215] [<ffffffc000929384>] tegra186_timer_isr+0x24/0x30
[  204.146979] [<ffffffc0000f5650>] handle_irq_event_percpu+0x84/0x290
[  204.153256] [<ffffffc0000f58a0>] handle_irq_event+0x44/0x74
[  204.158844] [<ffffffc0000f8ba8>] handle_fasteoi_irq+0xb4/0x188
[  204.164687] [<ffffffc0000f4c70>] generic_handle_irq+0x24/0x38
[  204.170443] [<ffffffc0000f4f78>] __handle_domain_irq+0x60/0xb4
[  204.176292] [<ffffffc000081774>] gic_handle_irq+0x5c/0xb4
[  204.181707] [<ffffffc000084740>] el1_irq+0x80/0xf8
[  204.186526] [<ffffffc000813838>] cpuidle_enter+0x18/0x20
[  204.191854] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[  204.197093] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[  204.202944] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[  204.209221] [<0000000080081acc>] 0x80081acc
[  240.346406] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[  240.355226] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[  240.361160] Modules linked in: tw6869(O) fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd bluedroid_pm
[  240.379737] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G        W  O    4.4.38 #5
[  240.386885] Hardware name: quill (DT)
[  240.390566] task: ffffffc1ece83e80 ti: ffffffc1ece9c000 task.ti: ffffffc1ece9c000
[  240.398057] PC is at 0x0
[  240.400617] LR is at t18x_a57_enter_state+0x20/0xc4
[  240.405507] pc : [<0000000000000000>] lr : [<ffffffc000941a04>] pstate: 800000c5
[  240.412906] sp : ffffffc1ece9fec0
[  240.416230] x29: ffffffc1ece9fec0 x28: ffffffc1ece9c000 
[  240.421581] x27: ffffffc000b74f00 x26: 00000037fe70ec82 
[  240.426931] x25: ffffffc0013f3000 x24: 0000000000000000 
[  240.432279] x23: ffffffc00137e048 x22: ffffffc00137e060 
[  240.437626] x21: ffffffc0013f31c8 x20: ffffffc0014bf720 
[  240.442973] x19: 0000000000000000 x18: 0000000000000a03 
[  240.448321] x17: 0000000000ad4878 x16: ffffffc00011a8fc 
[  240.453668] x15: 000000009837f050 x14: 0000000000000000 
[  240.459015] x13: 000000009837f050 x12: 000000000000d800 
[  240.464362] x11: 0000000000000400 x10: 00000000000008b0 
[  240.469709] x9 : ffffffc1ece9fec0 x8 : ffffffc1ece84790 
[  240.475055] x7 : 00000037de61ab60 x6 : 0000000000009074 
[  240.480401] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[  240.485747] x3 : 000000003b9aca00 x2 : 00000000003cde1e 
[  240.491095] x1 : 0000000000000000 x0 : 0000000000000000 
[  240.496439] 
[  240.497948] Process swapper/5 (pid: 0, stack limit = 0xffffffc1ece9c020)
[  240.504654] Call trace:
[  240.507115] [<          (null)>]           (null)
[  240.511845] [<ffffffc0008135ac>] cpuidle_enter_state+0x88/0x2dc
[  240.517780] [<ffffffc000813838>] cpuidle_enter+0x18/0x20
[  240.523108] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[  240.528347] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[  240.534198] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[  240.540472] [<0000000080081acc>] 0x80081acc
[  240.545448] ---[ end trace 937c384bd811378b ]---
[  240.557806] Kernel panic - not syncing: Attempted to kill the idle task!
[  241.780475] SMP: failed to stop secondary CPUs
[  241.821064] Rebooting in 5 seconds..
[  248.042147] SMP: failed to stop secondary CPUs

We found a strange problem with t18x_a57_enter_state function.
In the kernel log we found PC is at 0, LR is at t18x_a57_enter_state+0x20/0xc4 and x0=0000000000000000.
We do aarch64-linux-gnu-objdump -S t18x/drivers/cpuidle/cpuidle-tegra18x.0 > cpuidle-tegra18x.s.

0000000000000f9c <t18x_a57_enter_state>:
     f9c:       a9bd7bfd        stp     x29, x30, [sp,#-48]!
     fa0:       910003fd        mov     x29, sp
     fa4:       a90153f3        stp     x19, x20, [sp,#16]
     fa8:       f90013f5        str     x21, [sp,#32]
     fac:       2a0203f3        mov     w19, w2
     fb0:       94000000        bl      0 <tick_nohz_get_sleep_length>
     fb4:       94000000        bl      0 <ns_to_timespec>
     fb8:       94000000        bl      0 <tegra_cpu_is_asim>
     fbc:       53001c01        uxtb    w1, w0
     fc0:       2a0003e5        mov     w5, w0
     fc4:       350000e1        cbnz    w1, fe0 <t18x_a57_enter_state+0x44>
     fc8:       90000014        adrp    x20, 0 <denver_cc_get>
     fcc:       91000294        add     x20, x20, #0x0
     fd0:       b9402680        ldr     w0, [x20,#36]
     fd4:       35000120        cbnz    w0, ff8 <t18x_a57_enter_state+0x5c>
     fd8:       7100067f        cmp     w19, #0x1
     fdc:       54000380        b.eq    104c <t18x_a57_enter_state+0xb0>
     fe0:       d503207f        wfi
     fe4:       2a1303e0        mov     w0, w19
     fe8:       f94013f5        ldr     x21, [sp,#32]
     fec:       a94153f3        ldp     x19, x20, [sp,#16]
     ff0:       a8c37bfd        ldp     x29, x30, [sp],#48
     ff4:       d65f03c0        ret
     ff8:       b9405a80        ldr     w0, [x20,#88]
     ffc:       2a0103e2        mov     w2, w1
    1000:       2a0103e3        mov     w3, w1
    1004:       2a0103e4        mov     w4, w1
    1008:       90000015        adrp    x21, 0 <denver_cc_get>
    100c:       94000000        bl      0 <t18x_mce_update_cstate_info>
    1010:       910002b5        add     x21, x21, #0x0
    1014:       f9402a80        ldr     x0, [x20,#80]
    1018:       b987c2a1        ldrsw   x1, [x21,#1984]
    101c:       2a0003f3        mov     w19, w0
    1020:       eb00003f        cmp     x1, x0
    1024:       910fa2b5        add     x21, x21, #0x3e8
    1028:       54fffd88        b.hi    fd8 <t18x_a57_enter_state+0x3c>
    102c:       90000001        adrp    x1, 0 <denver_cc_get>
    1030:       90000000        adrp    x0, 0 <denver_cc_get>
    1034:       91000021        add     x1, x1, #0x0
    1038:       91000000        add     x0, x0, #0x0
    103c:       9133a021        add     x1, x1, #0xce8

LR is at instruction:

fbc:       53001c01        uxtb    w1, w0

PC is at 0, we think that the following instruction cause kernel crash

fb8:       94000000        bl      0 <tegra_cpu_is_asim>

But we do cat /proc/kallsyms | grep tegra_cpu_is_asim, we got

ffffffc000467c74 T tegra_cpu_is_asim

The tegra_cpu_is_asim symbol is not zero, why the instruction bl 0 <tegra_cpu_is_asim> cause kernel crash with PC is at 0?

We aslo did the follwing test without install tw6869 driver

sudo cat /dev/watchdog

The kernel log also print PC is at 0 in the t18x_a57_enter_state function:

nvidia@tegra-ubuntu:~$ [ 1331.658288] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[ 1331.658297] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[ 1331.658305] Bad mode in Synchronous Abort handler detected, code 0x86000005 -- IABT (current EL)
[ 1331.658314] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[ 1331.658320] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[ 1331.658392] Modules linked in: tw6869(O) ipt_MASQUERADE nf_nat_masquerade_ipv4 fuse iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd bluedroid_pm [last unloaded: timer_driver]
[ 1331.658404] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O    4.4.38 #7
[ 1331.658409] Hardware name: quill (DT)
[ 1331.658417] task: ffffffc1ece82580 ti: ffffffc1ece94000 task.ti: ffffffc1ece94000
[ 1331.658426] PC is at 0x0
[ 1331.658444] LR is at t18x_a57_enter_state+0x2c/0xe0
[ 1331.658450] pc : [<0000000000000000>] lr : [<ffffffc000941a10>] pstate: 800000c5
[ 1331.658454] sp : ffffffc1ece97ec0
[ 1331.658463] x29: ffffffc1ece97ec0 x28: ffffffc1ece94000 
[ 1331.658470] x27: ffffffc000b74f00 x26: 000001360dd30449 
[ 1331.658478] x25: ffffffc0013f3000 x24: 0000000000000000 
[ 1331.658485] x23: ffffffc00137e048 x22: ffffffc00137e060 
[ 1331.658493] x21: ffffffc0013f31c8 x20: ffffffc0014bf720 
[ 1331.658499] x19: 0000000000000000 x18: 0000000000000000 
[ 1331.658506] x17: 0000007f9edf4f68 x16: ffffffc000b75a60 
[ 1331.658512] x15: 00000000fa83b2da x14: 0000000000237694 
[ 1331.658519] x13: 0000000000000000 x12: 0000000000237694 
[ 1331.658525] x11: 0000000000000400 x10: 00000000000008b0 
[ 1331.658532] x9 : 000000010003ef7e x8 : ffffffc1ece82e90 
[ 1331.658539] x7 : 0000000000000000 x6 : 000000000002af86 
[ 1331.658545] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[ 1331.658552] x3 : 000000003b9aca00 x2 : 0000000011a7eef5 
[ 1331.658558] x1 : 0000000000000000 x0 : 0000000000000000 
[ 1331.658559] 
[ 1331.658566] Process swapper/3 (pid: 0, stack limit = 0xffffffc1ece94020)
[ 1331.658569] Call trace:
[ 1331.658575] [<          (null)>]           (null)
[ 1331.658596] [<ffffffc0008135ac>] cpuidle_enter_state+0x88/0x2dc
[ 1331.658605] [<ffffffc000813838>] cpuidle_enter+0x18/0x20
[ 1331.658621] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[ 1331.658628] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[ 1331.658642] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[ 1331.658647] [<0000000080081acc>] 0x80081acc
[ 1331.658659] Internal error: Oops - bad mode: 0 [#2] PREEMPT SMP
[ 1331.658699] Modules linked in: tw6869(O) ipt_MASQUERADE nf_nat_masquerade_ipv4 fuse iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd bluedroid_pm [last unloaded: timer_driver]
[ 1331.658709] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D    O    4.4.38 #7
[ 1331.658712] Hardware name: quill (DT)
[ 1331.658719] task: ffffffc1ece83200 ti: ffffffc1ece98000 task.ti: ffffffc1ece98000
[ 1331.658724] PC is at 0x0
[ 1331.658733] LR is at t18x_a57_enter_state+0x2c/0xe0
[ 1331.658739] pc : [<0000000000000000>] lr : [<ffffffc000941a10>] pstate: 800000c5
[ 1331.658742] sp : ffffffc1ece9bec0
[ 1331.658751] x29: ffffffc1ece9bec0 x28: ffffffc1ece98000 
[ 1331.658759] x27: ffffffc000b74f00 x26: 0000013616a97dd9 
[ 1331.658766] x25: ffffffc0013f3000 x24: 0000000000000000 
[ 1331.658773] x23: ffffffc00137e048 x22: ffffffc00137e060 
[ 1331.658780] x21: ffffffc0013f31c8 x20: ffffffc0014bf720 
[ 1331.658787] x19: 0000000000000000 x18: 0000005561d4f83c 
[ 1331.658793] x17: ffffffc000b75a60 x16: 0000000000000004 
[ 1331.658800] x15: 0000000000001efe x14: 000001932cbb7b80 
[ 1331.658807] x13: 0000000000000afb x12: 0000000000000400 
[ 1331.658813] x11: 0000000000000533 x10: 00000000000008b0 
[ 1331.658820] x9 : 000000010003efa4 x8 : ffffffc1ece83b10 
[ 1331.658827] x7 : 0000000000000000 x6 : 000000000002af96 
[ 1331.658834] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[ 1331.658840] x3 : 000000003b9aca00 x2 : 00000000031890a5 
[ 1331.658846] x1 : 0000000000000000 x0 : 0000000000000000 
[ 1331.658848] 
[ 1331.658854] Process swapper/4 (pid: 0, stack limit = 0xffffffc1ece98020)
[ 1331.658856] Call trace:
[ 1331.658860] [<          (null)>]           (null)
[ 1331.658873] [<ffffffc0008135ac>] cpuidle_enter_state+0x88/0x2dc
[ 1331.658882] [<ffffffc000813838>] cpuidle_enter+0x18/0x20
[ 1331.658890] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[ 1331.658897] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[ 1331.658905] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[ 1331.658909] [<0000000080081acc>] 0x80081acc
[ 1331.658919] Internal error: Oops - bad mode: 0 [#3] PREEMPT SMP
[ 1331.658958] Modules linked in: tw6869(O) ipt_MASQUERADE nf_nat_masquerade_ipv4 fuse iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd bluedroid_pm [last unloaded: timer_driver]
[ 1331.658968] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D    O    4.4.38 #7
[ 1331.658972] Hardware name: quill (DT)
[ 1331.658978] task: ffffffc001276240 ti: ffffffc001264000 task.ti: ffffffc001264000
[ 1331.658983] PC is at 0x0
[ 1331.658994] LR is at t18x_a57_enter_state+0x2c/0xe0
[ 1331.658999] pc : [<0000000000000000>] lr : [<ffffffc000941a10>] pstate: 800000c5
[ 1331.659002] sp : ffffffc001267e70
[ 1331.659012] x29: ffffffc001267e70 x28: ffffffc001264000 
[ 1331.659020] x27: ffffffc000b74f00 x26: 0000013617d9ecb3 
[ 1331.659028] x25: ffffffc0013f3000 x24: 0000000000000000 
[ 1331.659035] x23: ffffffc00137e048 x22: ffffffc00137e060 
[ 1331.659041] x21: ffffffc0013f31c8 x20: ffffffc0014bf720 
[ 1331.659048] x19: 0000000000000000 x18: 0000000000000a03 
[ 1331.659055] x17: 000000000230f9f8 x16: ffffffc000b75a60 
[ 1331.659061] x15: 00000000fa83b2da x14: 000000000000552a 
[ 1331.659068] x13: 0000000000000000 x12: 000000000000552a 
[ 1331.659074] x11: 0000000000000400 x10: 00000000000008b0 
[ 1331.659081] x9 : 000000010003efa3 x8 : ffffffc001276b50 
[ 1331.659088] x7 : 0000000000000000 x6 : 000000000002af98 
[ 1331.659094] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[ 1331.659100] x3 : 000000003b9aca00 x2 : 0000000001a7968f 
[ 1331.659106] x1 : 0000000000000000 x0 : 0000000000000000 
[ 1331.659108] 
[ 1331.659115] Process swapper/0 (pid: 0, stack limit = 0xffffffc001264020)
[ 1331.659117] Call trace:
[ 1331.659120] [<          (null)>]           (null)
[ 1331.659133] [<ffffffc0008135ac>] cpuidle_enter_state+0x88/0x2dc
[ 1331.659143] [<ffffffc000813838>] cpuidle_enter+0x18/0x20
[ 1331.659151] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[ 1331.659158] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[ 1331.659173] [<ffffffc000b649a8>] rest_init+0x84/0x8c
[ 1331.659185] [<ffffffc001120980>] start_kernel+0x3a0/0x3b4
[ 1331.659189] [<0000000080b6b000>] 0x80b6b000
[ 1331.659472] ---[ end trace 64e0eb23eec106a4 ]---
[ 1331.663804] ---[ end trace 64e0eb23eec106a5 ]---
[ 1331.663808] ---[ end trace 64e0eb23eec106a6 ]---
[ 1331.664365] Kernel panic - not syncing: Attempted to kill the idle task!
[ 1332.290368] Internal error: Oops - bad mode: 0 [#4] PREEMPT SMP
[ 1332.296290] Modules linked in: tw6869(O) ipt_MASQUERADE nf_nat_masquerade_ipv4 fuse iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd bluedroid_pm [last unloaded: timer_driver]
[ 1332.317412] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G      D    O    4.4.38 #7
[ 1332.324547] Hardware name: quill (DT)
[ 1332.328219] task: ffffffc1ece83e80 ti: ffffffc1ece9c000 task.ti: ffffffc1ece9c000
[ 1332.335703] PC is at 0x0
[ 1332.338253] LR is at t18x_a57_enter_state+0x2c/0xe0
[ 1332.343136] pc : [<0000000000000000>] lr : [<ffffffc000941a10>] pstate: 800000c5
[ 1332.350528] sp : ffffffc1ece9fec0
[ 1332.353848] x29: ffffffc1ece9fec0 x28: ffffffc1ece9c000 
[ 1332.359192] x27: ffffffc000b74f00 x26: 000001360df9aa92 
[ 1332.364533] x25: ffffffc0013f3000 x24: 0000000000000000 
[ 1332.369875] x23: ffffffc00137e048 x22: ffffffc00137e060 
[ 1332.375217] x21: ffffffc0013f31c8 x20: ffffffc0014bf720 
[ 1332.380557] x19: 0000000000000000 x18: 0000000000000a03 
[ 1332.385896] x17: ffffffc000b75a60 x16: ffffffc000b75a60 
[ 1332.391237] x15: 00000000fa83b2da x14: 0000000000003da4 
[ 1332.396573] x13: 0000000000000000 x12: 0000000000003da4 
[ 1332.401912] x11: 0000000000000400 x10: 00000000000008b0 
[ 1332.407250] x9 : 000000010003eeb6 x8 : ffffffc1ece84790 
[ 1332.412590] x7 : 0000000000000000 x6 : 000000000002af88 
[ 1332.417928] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[ 1332.423268] x3 : 000000003b9aca00 x2 : 00000000195f606d 
[ 1332.428606] x1 : 0000000000000000 x0 : 0000000000000000 
[ 1332.433944] 
[ 1332.435445] Process swapper/5 (pid: 0, stack limit = 0xffffffc1ece9c020)
[ 1332.442147] Call trace:
[ 1332.444601] [<          (null)>]           (null)
[ 1332.449319] [<ffffffc0008135ac>] cpuidle_enter_state+0x88/0x2dc
[ 1332.455246] [<ffffffc000813838>] cpuidle_enter+0x18/0x20
[ 1332.460564] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[ 1332.465795] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[ 1332.471635] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[ 1332.477903] [<0000000080081acc>] 0x80081acc
[ 1332.482433] ---[ end trace 64e0eb23eec106a7 ]---
[ 1332.871432] SMP: failed to stop secondary CPUs
[ 1332.900520] Rebooting in 5 seconds..
[ 1339.112134] SMP: failed to stop secondary CPUs
nvidia@tegra-ubuntu:~$ ls /dev/watchdog*
/dev/watchdog  /dev/watchdog0
nvidia@tegra-ubuntu:~$ zcat /proc/config.gz | grep WATCHDOG
# CONFIG_DPM_WATCHDOG is not set
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
# CONFIG_WATCHDOG_SYSFS is not set
# CONFIG_SOFT_WATCHDOG is not set
CONFIG_SOFT_PLATFORM_WATCHDOG=y
# CONFIG_GPIO_WATCHDOG is not set
# CONFIG_XILINX_WATCHDOG is not set
# CONFIG_ARM_SP805_WATCHDOG is not set
# CONFIG_CADENCE_WATCHDOG is not set
# CONFIG_DW_WATCHDOG is not set
# CONFIG_MAX63XX_WATCHDOG is not set
# CONFIG_MAX77620_WATCHDOG is not set
# CONFIG_TEGRA_WATCHDOG is not set
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_USBPCWATCHDOG is not set
CONFIG_TEGRA18X_WATCHDOG=y
# CONFIG_FIQ_WATCHDOG is not set

We want to know why the t18x_a57_enter_state fucnion could cause PC is at 0 error, thinks.

  1. This is normal behavior in kernel for “sudo cat /dev/watchdog”.

Once watchdog file is opened, watchdog need to be feed within 30 sec’s to avoid timeout. Otherwise, system will reboot.
If we keep opening this file or writing to it within 30 sec’s it never crashes and reboots.
e.g:

while true; do cat /dev/watchdog; sleep 15; done

Also as “# CONFIG_WATCHDOG_NOWAYOUT is not set”, we can close the file and disable watchdog to avoid reboot.

cat /dev/watchdog

cat: /dev/watchdog: Invalid argument
[ 183.146776] watchdog watchdog0: watchdog did not stop!

echo ‘V’ > /dev/watchdog

  1. It could be tw6869 driver cause system hang first and this message show up.

Hi ShaneCCC
Thank you very much for your reply.We want to find the last instruction executed before the system hang, but the kernel backtrace make us confused. Now we were confused about the backtrace information of watchdog.Why it print out PC is at 0 which seems to be unreasonable.Could you give us some guidance thanks !

Try to repo the problem with “sudo cat /dev/watchdog” but didn’t get same prints as mentioned.

root@jetson-P3310B00P0068:/home/ubuntu# cat /dev/watchdog
cat: /dev/watchdog: Invalid argument
[ 507.062728] watchdog watchdog0: watchdog did not stop!
root@jetson-P3310B00P0068:/home/ubuntu#
root@jetson-P3310B00P0068:/home/ubuntu# ASSERT: tegra_fiq_interrupt_handler <86> : ns_fiq_handler_addr
[0000.196] I> Welcome to MB2(TBoot-BPMP)(version: 01.00.160913-t186-M-00.00-mobile-20890243)

Hi ShaneCCC
Thank you very much for your reply.We download the jetpack3.2.1 and flash the tx2 module again. The problem is still the same.Command uname -a shows 4.4.38-tegra. Could you tell us your jetpack version? Thanks a lot!

nvidia@tegra-ubuntu:~$ uname -a
Linux tegra-ubuntu 4.4.38-tegra #1 SMP PREEMPT Thu May 17 00:15:19 PDT 2018 aarch64 aarch64 aarch64 GNU/Linux
nvidia@tegra-ubuntu:~$ sudo cat /dev/watchdog
[sudo] password for nvidia: 
cat: /dev/watchdog: Invalid argume[  153.711246] watchdog watchdog0: watchdog did not stop!
nt
nvidia@tegra-ubuntu:~$ [  243.719431] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[  243.719444] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[  243.719458] Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
[  243.719472] Bad mode in Synchronous Abort handler detected, code 0x86000005 -- IABT (current EL)
[  243.719483] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[  243.719643] Modules linked in: fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd pci_tegra bluedroid_pm
[  243.719674] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.4.38-tegra #1
[  243.719678] Hardware name: quill (DT)
[  243.719694] task: ffffffc1ece83e80 ti: ffffffc1ecea0000 task.ti: ffffffc1ecea0000
[  243.719712] PC is at 0x0
[  243.719784] LR is at t18x_a57_enter_state+0x20/0xc4
[  243.719792] pc : [<0000000000000000>] lr : [<ffffffc00094eef4>] pstate: 800000c5
[  243.719796] sp : ffffffc1ecea3ec0
[  243.719810] x29: ffffffc1ecea3ec0 x28: ffffffc1ecea0000 
[  243.719821] x27: ffffffc000b87f00 x26: 00000038bb4959fd 
[  243.719833] x25: ffffffc00140d000 x24: 0000000000000000 
[  243.719843] x23: ffffffc001396b48 x22: ffffffc001396b60 
[  243.719853] x21: ffffffc00140d6c8 x20: ffffffc0014d8bb8 
[  243.719863] x19: 0000000000000000 x18: 0000000000000a03 
[  243.719872] x17: 00000000023105a0 x16: ffffffc00011a8fc 
[  243.719882] x15: ffffffc000b88a60 x14: 0000000000000001 
[  243.719892] x13: 0000000000005d14 x12: 000039257cf38800 
[  243.719901] x11: 0000000000000000 x10: 0000000000000040 
[  243.719912] x9 : 0000000000ffffc9 x8 : 0000000000000009 
[  243.719921] x7 : 0000000000000009 x6 : 0000000000010006 
[  243.719930] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[  243.719940] x3 : 000000003b9aca00 x2 : 0000000005f15303 
[  243.719949] x1 : 0000000000000000 x0 : 0000000000000000 
[  243.719951] 
[  243.719962] Process swapper/5 (pid: 0, stack limit = 0xffffffc1ecea0020)
[  243.719970] Call trace:
[  243.719980] [<          (null)>]           (null)
[  243.720031] [<ffffffc000820a94>] cpuidle_enter_state+0x88/0x2dc
[  243.720043] [<ffffffc000820d20>] cpuidle_enter+0x18/0x20
[  243.720082] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[  243.720096] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[  243.720127] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[  243.720134] [<0000000080081acc>] 0x80081acc
[  243.720161] Internal error: Oops - bad mode: 0 [#2] PREEMPT SMP
[  243.720211] Modules linked in: fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd pci_tegra bluedroid_pm
[  243.720228] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D         4.4.38-tegra #1
[  243.720233] Hardware name: quill (DT)
[  243.720242] task: ffffffc1ece82580 ti: ffffffc1ece98000 task.ti: ffffffc1ece98000
[  243.720249] PC is at 0x0
[  243.720266] LR is at t18x_a57_enter_state+0x20/0xc4
[  243.720274] pc : [<0000000000000000>] lr : [<ffffffc00094eef4>] pstate: 800000c5
[  243.720278] sp : ffffffc1ece9bec0
[  243.720292] x29: ffffffc1ece9bec0 x28: ffffffc1ece98000 
[  243.720303] x27: ffffffc000b87f00 x26: 00000038b7816755 
[  243.720314] x25: ffffffc00140d000 x24: 0000000000000000 
[  243.720325] x23: ffffffc001396b48 x22: ffffffc001396b60 
[  243.720335] x21: ffffffc00140d6c8 x20: ffffffc0014d8bb8 
[  243.720345] x19: 0000000000000000 x18: 0000000000000a03 
[  243.720357] x17: 00000000023105c0 x16: ffffffc00011a8fc 
[  243.720368] x15: ffffffc000b88a60 x14: 0000000000000001 
[  243.720378] x13: 0000000000008451 x12: 000068c4ba692400 
[  243.720389] x11: 0000000000000000 x10: 00000000000008b0 
[  243.720399] x9 : 00000000ffffc8cd x8 : ffffffc1ece82e90 
[  243.720408] x7 : 0000000000000000 x6 : 000000000000fff4 
[  243.720417] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[  243.720427] x3 : 000000003b9aca00 x2 : 000000001c8f18ab 
[  243.720435] x1 : 0000000000000000 x0 : 0000000000000000 
[  243.720438] 
[  243.720447] Process swapper/3 (pid: 0, stack limit = 0xffffffc1ece98020)
[  243.720450] Call trace:
[  243.720455] [<          (null)>]           (null)
[  243.720471] [<ffffffc000820a94>] cpuidle_enter_state+0x88/0x2dc
[  243.720483] [<ffffffc000820d20>] cpuidle_enter+0x18/0x20
[  243.720493] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[  243.720502] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[  243.720515] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[  243.720524] [<0000000080081acc>] 0x80081acc
[  243.720540] Internal error: Oops - bad mode: 0 [#3] PREEMPT SMP
[  243.720592] Modules linked in: fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd pci_tegra bluedroid_pm
[  243.720607] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D         4.4.38-tegra #1
[  243.720613] Hardware name: quill (DT)
[  243.720624] task: ffffffc1ece83200 ti: ffffffc1ece9c000 task.ti: ffffffc1ece9c000
[  243.720635] PC is at 0x0
[  243.720652] LR is at t18x_a57_enter_state+0x20/0xc4
[  243.720661] pc : [<0000000000000000>] lr : [<ffffffc00094eef4>] pstate: 800000c5
[  243.720665] sp : ffffffc1ece9fec0
[  243.720679] x29: ffffffc1ece9fec0 x28: ffffffc1ece9c000 
[  243.720691] x27: ffffffc000b87f00 x26: 00000038bd77e558 
[  243.720702] x25: ffffffc00140d000 x24: 0000000000000000 
[  243.720713] x23: ffffffc001396b48 x22: ffffffc001396b60 
[  243.720724] x21: ffffffc00140d6c8 x20: ffffffc0014d8bb8 
[  243.720734] x19: 0000000000000000 x18: 0000000000000000 
[  243.720744] x17: 0000007fa164c938 x16: ffffffc0001e7c38 
[  243.720753] x15: ffffffc000b88a60 x14: 0000000000000001 
[  243.720763] x13: 00000000000162fd x12: 00003be8acddcdb3 
[  243.720773] x11: 0000000000000000 x10: 00000000000008b0 
[  243.720783] x9 : 00000000ffffc900 x8 : ffffffc1ece83b10 
[  243.720793] x7 : ffffffc1e0447880 x6 : 0000000000010010 
[  243.720802] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[  243.720811] x3 : 000000003b9aca00 x2 : 0000000000f3fecf 
[  243.720820] x1 : 0000000000000000 x0 : 0000000000000000 
[  243.720822] 
[  243.720831] Process swapper/4 (pid: 0, stack limit = 0xffffffc1ece9c020)
[  243.720834] Call trace:
[  243.720840] [<          (null)>]           (null)
[  243.720856] [<ffffffc000820a94>] cpuidle_enter_state+0x88/0x2dc
[  243.720868] [<ffffffc000820d20>] cpuidle_enter+0x18/0x20
[  243.720879] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[  243.720889] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[  243.720901] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[  243.720907] [<0000000080081acc>] 0x80081acc
[  243.724049] ---[ end trace 988fa02c861b9a1b ]---
[  243.730696] ---[ end trace 988fa02c861b9a1c ]---
[  243.730700] ---[ end trace 988fa02c861b9a1d ]---
[  243.731548] Kernel panic - not syncing: Attempted to kill the idle task!
[  244.345240] Internal error: Oops - bad mode: 0 [#4] PREEMPT SMP
[  244.351184] Modules linked in: fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd pci_tegra bluedroid_pm
[  244.369937] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D         4.4.38-tegra #1
[  244.377616] Hardware name: quill (DT)
[  244.381311] task: ffffffc00128a240 ti: ffffffc001278000 task.ti: ffffffc001278000
[  244.388818] PC is at 0x0
[  244.391405] LR is at t18x_a57_enter_state+0x20/0xc4
[  244.396309] pc : [<0000000000000000>] lr : [<ffffffc00094eef4>] pstate: 800000c5
[  244.403724] sp : ffffffc00127be70
[  244.407062] x29: ffffffc00127be70 x28: ffffffc001278000 
[  244.412452] x27: ffffffc000b87f00 x26: 00000038bd5b7563 
[  244.417839] x25: ffffffc00140d000 x24: 0000000000000000 
[  244.423225] x23: ffffffc001396b48 x22: ffffffc001396b60 
[  244.428613] x21: ffffffc00140d6c8 x20: ffffffc0014d8bb8 
[  244.433996] x19: 0000000000000000 x18: 0000000000000000 
[  244.439383] x17: 0000007f9b58ef18 x16: ffffffc000b88a60 
[  244.444765] x15: 00000000fa83b2da x14: 000000000000a563 
[  244.450147] x13: 0000000000000000 x12: 000000000000a563 
[  244.455530] x11: 0000000000000400 x10: 00000000000008b0 
[  244.460916] x9 : 00000000ffffc8ed x8 : ffffffc00128ab50 
[  244.466299] x7 : 0000000000000001 x6 : 0000000000010010 
[  244.471683] x5 : 0000000000000000 x4 : 00ffffffffffffff 
[  244.477065] x3 : 000000003b9aca00 x2 : 000000000735993d 
[  244.482447] x1 : 0000000000000000 x0 : 0000000000000000 
[  244.487824] 
[  244.489347] Process swapper/0 (pid: 0, stack limit = 0xffffffc001278020)
[  244.496067] Call trace:
[  244.498537] [<          (null)>]           (null)
[  244.503284] [<ffffffc000820a94>] cpuidle_enter_state+0x88/0x2dc
[  244.509232] [<ffffffc000820d20>] cpuidle_enter+0x18/0x20
[  244.514573] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[  244.519828] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[  244.525735] [<ffffffc000b7786c>] rest_init+0x84/0x8c
[  244.530765] [<ffffffc001132980>] start_kernel+0x3a0/0x3b4
[  244.536185] [<0000000080b7e000>] 0x80b7e000
[  244.541153] ---[ end trace 988fa02c861b9a1e ]---
[  245.071597] SMP: failed to stop secondary CPUs
[  245.110738] Rebooting in 5 seconds..
[  251.455841] SMP: failed to stop secondary CPUs
[0000.081] C> I2C command failed

@liukejob
I try on the internal r28.3, this version should be release soon.