TX1 board freeze

Charlinux · January 27, 2016, 10:39am

Hi,

I’m using the TX1 board with WebRTC for audio/video conferencing with a web browser.

Sometimes, the board freezes during a call. Roughly, it happens 5 times a day.
The GPU is encoding two video streams in VP8 and decoding one VP8 stream.
The CPU seems under quite a heavy load:

Load average: 5.78 4.63 2.671

With a serial port plugged into the board, I managed to get some kernel logs on two different TX1 board:

[ 5869.738665] end_request: I/O error, dev mmcblk0, sector 9886242
[ 5869.744637] end_request: I/O error, dev mmcblk0, sector 9886250
[ 5869.750578] end_request: I/O error, dev mmcblk0, sector 9886258
[ 5869.756536] end_request: I/O error, dev mmcblk0, sector 9886266
[ 5869.762462] end_request: I/O error, dev mmcblk0, sector 9886274
[ 5869.768410] end_request: I/O error, dev mmcblk0, sector 9886282
[ 5869.774340] end_request: I/O error, dev mmcblk0, sector 9886290
[ 5869.780257] end_request: I/O error, dev mmcblk0, sector 9886298
[ 5869.786189] end_request: I/O error, dev mmcblk0, sector 9886306
[ 5869.792104] end_request: I/O error, dev mmcblk0, sector 9886314
[ 5873.684006] mmc1: Timeout waiting for hardware interrupt.
[ 5873.689397] sdhci: ================== REGISTER DUMP (mmc1)==================
[ 5873.696433] sdhci: Sys addr[0x000]: 0x00000000 | Version[0x0fe]:  0x00000303
[ 5873.703468] sdhci: Blk size[0x004]: 0x00007080 | Blk cnt[0x006]:  0x00000000
[ 5873.710503] sdhci: Argument[0x008]: 0x92003e00 | Trn mode[0x00c]: 0x00000000
[ 5873.717537] sdhci: Present[0x024]:  0x01fb00f0 | Host ctl[0x028]: 0x00000013
[ 5873.724571] sdhci: Power[0x029]:    0x0000000d | Blk gap[0x02a]:  0x00000000
[ 5873.731605] sdhci: Wake-up[0x02b]:  0x00000000 | Clock[0x02c]:    0x00000007
[ 5873.738637] sdhci: Timeout[0x02e]:  0x0000000e | Int stat[0x030]: 0x00000001
[ 5873.745672] sdhci: Int enab[0x034]: 0x02ff000b | Sig enab[0x038]: 0x02fc000b
[ 5873.752705] sdhci: AC12 err[0x03c]: 0x00000000 | Slot int[0x0fc]: 0x00000000
[ 5873.759740] sdhci: Caps[0x040]:     0x376cd08c | Caps_1[0x044]:   0x10006f73
[ 5873.766775] sdhci: Cmd[0x00e]:      0x0000341a | Max curr[0x048]: 0x00000000
[ 5873.773807] sdhci: Host ctl2[0x03e]: 0x0000300b
[ 5873.778328] sdhci: ADMA Err[0x054]: 0x00000000 | ADMA Ptr[0x058]: 0xe9500010
[ 5873.785413] sdhci: =========================================================
[ 5879.824075] mmc0: Timeout waiting for hardware interrupt.
[ 5879.829466] sdhci: ================== REGISTER DUMP (mmc0)==================
[ 5879.836503] sdhci: Sys addr[0x000]: 0x00000000 | Version[0x0fe]:  0x00000303
[ 5879.843537] sdhci: Blk size[0x004]: 0x00007200 | Blk cnt[0x006]:  0x00000000
[ 5879.850572] sdhci: Argument[0x008]: 0x00000018 | Trn mode[0x00c]: 0x00000000
[ 5879.857606] sdhci: Present[0x024]:  0x01fb00f0 | Host ctl[0x028]: 0x00000030
[ 5879.864639] sdhci: Power[0x029]:    0x0000000b | Blk gap[0x02a]:  0x00000000
[ 5879.871674] sdhci: Wake-up[0x02b]:  0x00000000 | Clock[0x02c]:    0x00000007
[ 5879.878707] sdhci: Timeout[0x02e]:  0x0000000e | Int stat[0x030]: 0x00000001
[ 5879.885741] sdhci: Int enab[0x034]: 0x02ff000b | Sig enab[0x038]: 0x02fc000b
[ 5879.892775] sdhci: AC12 err[0x03c]: 0x00000000 | Slot int[0x0fc]: 0x00000000
[ 5879.899810] sdhci: Caps[0x040]:     0x376cd08c | Caps_1[0x044]:   0x10006f77
[ 5879.906843] sdhci: Cmd[0x00e]:      0x0000171a | Max curr[0x048]: 0x00000000
[ 5879.913875] sdhci: Host ctl2[0x03e]: 0x0000308d
[ 5879.918395] sdhci: ADMA Err[0x054]: 0x00000000 | ADMA Ptr[0x058]: 0x80000010
[ 5879.925478] sdhci: =========================================================
[ 5883.804118] mmc1: Timeout waiting for hardware interrupt.
[ 5883.809508] sdhci: ================== REGISTER DUMP (mmc1)==================
[ 5883.816544] sdhci: Sys addr[0x000]: 0x00000000 | Version[0x0fe]:  0x00000303
[ 5883.823580] sdhci: Blk size[0x004]: 0x00007080 | Blk cnt[0x006]:  0x00000000
[ 5883.830615] sdhci: Argument[0x008]: 0x92003e00 | Trn mode[0x00c]: 0x00000000
[ 5883.837651] sdhci: Present[0x024]:  0x01fb00f0 | Host ctl[0x028]: 0x00000013
[ 5883.844685] sdhci: Power[0x029]:    0x0000000d | Blk gap[0x02a]:  0x00000000
[ 5883.851718] sdhci: Wake-up[0x02b]:  0x00000000 | Clock[0x02c]:    0x00000007
[ 5883.858752] sdhci: Timeout[0x02e]:  0x0000000e | Int stat[0x030]: 0x00000001
[ 5883.865787] sdhci: Int enab[0x034]: 0x02ff000b | Sig enab[0x038]: 0x02fc000b
[ 5883.872821] sdhci: AC12 err[0x03c]: 0x00000000 | Slot int[0x0fc]: 0x00000000
[ 5883.879856] sdhci: Caps[0x040]:     0x376cd08c | Caps_1[0x044]:   0x10006f73
[ 5883.886890] sdhci: Cmd[0x00e]:      0x0000341a | Max curr[0x048]: 0x00000000
[ 5883.893923] sdhci: Host ctl2[0x03e]: 0x0000300b
[ 5883.898444] sdhci: ADMA Err[0x054]: 0x00000000 | ADMA Ptr[0x058]: 0xe9500010
[ 5883.905528] sdhci: =========================================================
[ 5889.944188] mmc0: Timeout waiting for hardware interrupt.
[ 5889.949578] sdhci: ================== REGISTER DUMP (mmc0)==================
[ 5889.956615] sdhci: Sys addr[0x000]: 0x00000000 | Version[0x0fe]:  0x00000303
[ 5889.963650] sdhci: Blk size[0x004]: 0x00007200 | Blk cnt[0x006]:  0x00000000
[ 5889.970683] sdhci: Argument[0x008]: 0x00010000 | Trn mode[0x00c]: 0x00000000
[ 5889.977717] sdhci: Present[0x024]:  0x01fb00f0 | Host ctl[0x028]: 0x00000031
[ 5889.984752] sdhci: Power[0x029]:    0x0000000b | Blk gap[0x02a]:  0x00000000
[ 5889.991785] sdhci: Wake-up[0x02b]:  0x00000000 | Clock[0x02c]:    0x00000007
[ 5889.998819] sdhci: Timeout[0x02e]:  0x0000000e | Int stat[0x030]: 0x00000001
[ 5890.005853] sdhci: Int enab[0x034]: 0x02ff000b | Sig enab[0x038]: 0x02fc000b
[ 5890.012886] sdhci: AC12 err[0x03c]: 0x00000000 | Slot int[0x0fc]: 0x00000000
[ 5890.019921] sdhci: Caps[0x040]:     0x376cd08c | Caps_1[0x044]:   0x10006f77
[ 5890.026954] sdhci: Cmd[0x00e]:      0x00000d1a | Max curr[0x048]: 0x00000000
[ 5890.033987] sdhci: Host ctl2[0x03e]: 0x0000308d
[ 5890.038507] sdhci: ADMA Err[0x054]: 0x00000000 | ADMA Ptr[0x058]: 0x80000010
[ 5890.045593] sdhci: =========================================================
[ 5890.052816] mmcblk0: error -110 sending status command, retrying
[ 5893.924233] mmc1: Timeout waiting for hardware interrupt.
[ 5893.929623] sdhci: ================== REGISTER DUMP (mmc1)==================
[ 5893.936659] sdhci: Sys addr[0x000]: 0x00000000 | Version[0x0fe]:  0x00000303
[ 5893.943695] sdhci: Blk size[0x004]: 0x00007080 | Blk cnt[0x006]:  0x00000000
[ 5893.950730] sdhci: Argument[0x008]: 0x12003e00 | Trn mode[0x00c]: 0x00000000
[ 5893.957765] sdhci: Present[0x024]:  0x01fb00f0 | Host ctl[0x028]: 0x00000013
[ 5893.964800] sdhci: Power[0x029]:    0x0000000d | Blk gap[0x02a]:  0x00000000
[ 5893.971835] sdhci: Wake-up[0x02b]:  0x00000000 | Clock[0x02c]:    0x00000007
[ 5893.978867] sdhci: Timeout[0x02e]:  0x0000000e | Int stat[0x030]: 0x00000001
[ 5893.985902] sdhci: Int enab[0x034]: 0x02ff000b | Sig enab[0x038]: 0x02fc000b
[ 5893.992936] sdhci: AC12 err[0x03c]: 0x00000000 | Slot int[0x0fc]: 0x00000000
[ 5893.999970] sdhci: Caps[0x040]:     0x376cd08c | Caps_1[0x044]:   0x10006f73
[ 5894.007005] sdhci: Cmd[0x00e]:      0x0000341a | Max curr[0x048]: 0x00000000
[ 5894.014037] sdhci: Host ctl2[0x03e]: 0x0000300b
[ 5894.018558] sdhci: ADMA Err[0x054]: 0x00000000 | ADMA Ptr[0x058]: 0xe9500010
[ 5894.025644] sdhci: =========================================================

...

[ 5991.333924] mmcblk0: error -110 sending status command, aborting
[ 5991.340048] end_request: I/O error, dev mmcblk0, sector 1660562
[ 5991.346011] end_request: I/O error, dev mmcblk0, sector 1660570
[ 5991.351985] Buffer I/O error on device mmcblk0p1, logical block 207566
[ 5991.358546] Buffer I/O error on device mmcblk0p1, logical block 207567

and, on the second board:

[ 4757.081269] Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP
[ 4757.088314] Enter nvdumper_crash_setup_regs
[ 4757.092501] nvdumper: all registers are saved.
[ 4757.092504] nvdumper: all registers are saved.
[ 4757.092505] nvdumper: all registers are saved.
[ 4757.105826] nvdumper: all registers are saved.
[ 4757.110268] Modules linked in: rfcomm bnep bcmdhd cfg80211 nvhost_vi bluedroid_pm
[ 4757.117860] CPU: 0 PID: 934 Comm: kworker/0:1H Not tainted 3.10.67-g3a5c467 #1
[ 4757.125080] Workqueue: host_syncpt syncpt_thresh_cascade_fn
[ 4757.130662] task: ffffffc0fcda0ac0 ti: ffffffc0f2008000 task.ti: ffffffc0f2008000
[ 4757.138137] PC is at 0xffffffc0f427a8c0
[ 4757.141974] LR is at __wake_up_common+0x64/0xa4
[ 4757.146500] pc : [<ffffffc0f427a8c0>] lr : [<ffffffc0000d9398>] pstate: 800001c5
[ 4757.153885] sp : ffffffc0f200bba0
[ 4757.157198] x29: ffffffc0f200bba0 x28: ffffffc0f200bd28
[ 4757.162533] x27: ffffffc000f3d0d0 x26: 0000000000000010
[ 4757.167867] x25: 000000000046b2e8 x24: 0000000000000001
[ 4757.173205] x23: 0000000000000001 x22: 0000000000000000
[ 4757.178541] x21: 0000000000000000 x20: ffffffc0fb163db0
[ 4757.183877] x19: ffffffc0fb163e68 x18: 0000000000000000
[ 4757.189215] x17: 0000000000000000 x16: 0000000000000000
[ 4757.194550] x15: 0000000000000024 x14: 00000000000000de
[ 4757.199886] x13: 00000000000000b5 x12: 00000000000000f0
[ 4757.205217] x11: 00000000eeffed88 x10: 000000003edd4000
[ 4757.210552] x9 : ffffffc0f200bc20 x8 : ffffffc0fcda0fe0
[ 4757.215888] x7 : 0000000032d481ea x6 : ffffffc0ec34c3c0
[ 4757.221223] x5 : ffffffc0f427a8c0 x4 : 0000000000000000
[ 4757.226557] x3 : 0000000000000000 x2 : 0000000000000000
[ 4757.231895] x1 : 0000000000000001 x0 : ffffffc0fb163de8
[ 4757.237233]
[ 4757.237233] PC: 0xffffffc0f427a840:
[ 4757.242190] a840  09c20110 ffffff80 09c20000 ffffff80 00000000 00000000 00000001 00000000
[ 4757.250499] a860  f427a860 ffffffc0 f427a860 ffffffc0 00000000 00000000 00000000 00000000
[ 4757.258803] a880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.267115] a8a0  ffffffe0 0000000f f427a8a8 ffffffc0 f427a8a8 ffffffc0 004668c8 ffffffc0
[ 4757.275425] a8c0  fe438000 ffffffc0 00100100 00000000 00200200 00000000 b6e7b6e7 00000001
[ 4757.283732] a8e0  00000005 00000000 b6e2b6e2 00000000 f427a8f0 ffffffc0 f427a8f0 ffffffc0
[ 4757.292038] a900  000001fa 00000101 000002ce 00000000 00000001 00000000 f427a918 ffffffc0
[ 4757.300343] a920  f427a918 ffffffc0 00000000 00000000 00000000 00000000 ffffffff 00000000
[ 4757.308651]
[ 4757.308651] LR: 0xffffffc0000d9318:
[ 4757.313608] 9318  9b037c03 d347fc63 d341fc21 91000442 b5ffff41 aa0303e0 d65f03c0 a9bb7bfd
[ 4757.321915] 9338  910003fd a90153f3 a9025bf5 a90363f7 f90023f9 2a0103f7 2a0203f8 2a0303f6
[ 4757.330224] 9358  aa0403f5 aa0003f4 f8408e81 aa0103e0 f85e8413 d1006273 eb01029f 54000061
[ 4757.338534] 9378  14000012 aa0503f3 b9400019 f9400805 2a1703e1 2a1603e2 aa1503e3 d63f00a0
[ 4757.346840] 9398  34000080 36000079 71000718 540000e0 f9400e65 d10060a5 aa1303e0 91006273
[ 4757.355143] 93b8  eb13029f 54fffe01 a94153f3 a9425bf5 a94363f7 f94023f9 a8c57bfd d65f03c0
[ 4757.363449] 93d8  a9bf7bfd 910003fd 52800003 d2800004 97ffffd3 a8c17bfd d65f03c0 a9bf7bfd
[ 4757.371755] 93f8  910003fd aa0203e4 52800022 52800003 97ffffcb a8c17bfd d65f03c0 b9403400
[ 4757.380062]
[ 4757.380062] SP: 0xffffffc0f200bb20:
[ 4757.385021] bb20  fb163db0 ffffffc0 00000000 00000000 00000000 00000000 00000001 00000000
[ 4757.393329] bb40  00000001 00000000 0046b2e8 00000000 00000010 00000000 00f3d0d0 ffffffc0
[ 4757.401644] bb60  f200bd28 ffffffc0 f200bba0 ffffffc0 000d9398 ffffffc0 f200bba0 ffffffc0
[ 4757.409951] bb80  f427a8c0 ffffffc0 800001c5 00000000 00000000 00000000 00000001 00000000
[ 4757.418259] bba0  f200bbf0 ffffffc0 000dbc68 ffffffc0 fb163da8 ffffffc0 00000140 00000000
[ 4757.426566] bbc0  00000000 00000000 00000001 00000000 00000001 00000000 003732d0 ffffffc0
[ 4757.434875] bbe0  00373270 ffffffc0 f200bd18 ffffffc0 f200bc30 ffffffc0 00373324 ffffffc0
[ 4757.443183] bc00  fb163da8 ffffffc0 f200bd18 ffffffc0 003732d0 ffffffc0 f200bd18 ffffffc0
[ 4757.451490]
[ 4757.451490] X0: 0xffffffc0fb163d68:
[ 4757.456449] 3d68  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.464755] 3d88  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.473056] 3da8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.481364] 3dc8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.489672] 3de8  00000000 00000000 00000000 00000000 00000000 00000000 fb163e40 ffffffc0
[ 4757.497978] 3e08  001ad388 ffffffc0 fb163e40 ffffffc0 001ad398 ffffffc0 00000000 00000000
[ 4757.506285] 3e28  fb163ec0 ffffffc0 00001295 00000000 1c8d91c3 00000000 fb163d40 ffffffc0
[ 4757.514593] 3e48  fb163d60 ffffffc0 fb163d80 ffffffc0 fb163da0 ffffffc0 fb163dc0 ffffffc0
[ 4757.522903]
[ 4757.522903] X5: 0xffffffc0f427a840:
[ 4757.527859] a840  09c20110 ffffff80 09c20000 ffffff80 00000000 00000000 00000001 00000000
[ 4757.536162] a860  f427a860 ffffffc0 f427a860 ffffffc0 00000000 00000000 00000000 00000000
[ 4757.544469] a880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.552777] a8a0  ffffffe0 0000000f f427a8a8 ffffffc0 f427a8a8 ffffffc0 004668c8 ffffffc0
[ 4757.561086] a8c0  fe438000 ffffffc0 00100100 00000000 00200200 00000000 b6e7b6e7 00000001
[ 4757.569395] a8e0  00000005 00000000 b6e2b6e2 00000000 f427a8f0 ffffffc0 f427a8f0 ffffffc0
[ 4757.577702] a900  000001fa 00000101 000002ce 00000000 00000001 00000000 f427a918 ffffffc0
[ 4757.586008] a920  f427a918 ffffffc0 00000000 00000000 00000000 00000000 ffffffff 00000000
[ 4757.594314]
[ 4757.594314] X6: 0xffffffc0ec34c340:
[ 4757.599272] c340  e4b45300 ffffffc0 00005158 00000000 00000000 00000012 00000000 00000000
[ 4757.607583] c360  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.615885] c380  ec34c280 ffffffc0 00000000 636e6566 00000065 00000000 00000000 00000000
[ 4757.624189] c3a0  00000000 00000000 00000000 00000000 ec34c2c0 ffffffc0 ec34c2c0 ffffffc0
[ 4757.632494] c3c0  ec34c3c0 ffffffc0 ec34c3c0 ffffffc0 00020002 00000001 00010001 00000000
[ 4757.640799] c3e0  ec34c3e0 ffffffc0 ec34c3e0 ffffffc0 00100100 00000000 00200200 00000000
[ 4757.649109] c400  ec34c200 ffffffc0 0046c128 ffffffc0 0046c0d0 ffffffc0 0046c0ac ffffffc0
[ 4757.657418] c420  0046c080 ffffffc0 0046bd9c ffffffc0 0046bd04 ffffffc0 00000001 00000000
[ 4757.665726]
[ 4757.665726] X8: 0xffffffc0fcda0f60:
[ 4757.670687] 0f60  48313a30 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.678992] 0f80  fd682b00 ffffffc0 fcda0ac0 ffffffc0 fd682b00 ffffffc0 010e58c0 ffffffc0
[ 4757.687302] 0fa0  00000000 00000000 010e58c0 ffffffc0 010e58c0 ffffffc0 0130f848 ffffffc0
[ 4757.695609] 0fc0  0130f868 ffffffc0 fd544c00 ffffffc0 f200bc20 ffffffc0 f200bc20 ffffffc0
[ 4757.703916] 0fe0  00086adc ffffffc0 00000000 00000000 fec5b813 4177d783 6e617473 00006563
[ 4757.712226] 1000  2f747261 73626f6a 74656e2f 6b726f77 6964325f 7265746e cc59b264 329506bf
[ 4757.720534] 1020  00000000 7fffffff e6cd15c3 42174876 cc59b264 4177d784 00000000 408f4000
[ 4757.728842] 1040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.737148]
[ 4757.737148] X9: 0xffffffc0f200bba0:
[ 4757.742105] bba0  f200bbf0 ffffffc0 000dbc68 ffffffc0 fb163da8 ffffffc0 00000140 00000000
[ 4757.750414] bbc0  00000000 00000000 00000001 00000000 00000001 00000000 003732d0 ffffffc0
[ 4757.758722] bbe0  00373270 ffffffc0 f200bd18 ffffffc0 f200bc30 ffffffc0 00373324 ffffffc0
[ 4757.767030] bc00  fb163da8 ffffffc0 f200bd18 ffffffc0 003732d0 ffffffc0 f200bd18 ffffffc0
[ 4757.775341] bc20  00000100 00000000 f200bd28 ffffffc0 f200bc50 ffffffc0 00373124 ffffffc0
[ 4757.783647] bc40  e4916240 ffffffc0 00373180 ffffffc0 f200bcb0 ffffffc0 00373648 ffffffc0
[ 4757.791955] bc60  feec0320 ffffffc0 00000001 00000000 00000000 00000000 feec0330 ffffffc0
[ 4757.800263] bc80  fee9d210 ffffffc0 0015cf1e 00000000 00000000 00000000 3feb9500 ffffffc0
[ 4757.808579]
[ 4757.808579] X19: 0xffffffc0fb163de8:
[ 4757.813626] 3de8  00000000 00000000 00000000 00000000 00000000 00000000 fb163e40 ffffffc0
[ 4757.821931] 3e08  001ad388 ffffffc0 fb163e40 ffffffc0 001ad398 ffffffc0 00000000 00000000
[ 4757.830236] 3e28  fb163ec0 ffffffc0 00001295 00000000 2fc1f349 00000000 fb163d40 ffffffc0
[ 4757.838540] 3e48  fb163d60 ffffffc0 fb163d80 ffffffc0 fb163da0 ffffffc0 fb163dc0 ffffffc0
[ 4757.846845] 3e68  fb163de0 ffffffc0 00000000 00000000 000853c4 ffffffc0 00000000 00000000
[ 4757.855157] 3e88  00000000 00000000 ffffffff ffffffff f71a5958 00000000 60030030 00000000
[ 4757.863464] 3ea8  00000000 00000000 ffffffff ffffffff 00000000 00003e80 00001295 00000000
[ 4757.871769] 3ec8  31accf32 00000000 00000100 00000000 ab3a8270 00000000 00000000 00000000
[ 4757.880076]
[ 4757.880076] X20: 0xffffffc0fb163d30:
[ 4757.885122] 3d30  0008e000 ffffffc0 00000001 00000000 4fe2402a 00000006 00000000 00000000
[ 4757.893427] 3d50  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.901738] 3d70  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.910043] 3d90  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.918355] 3db0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.926662] 3dd0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4757.934967] 3df0  00000000 00000000 00000000 00000000 fb163e40 ffffffc0 001ad388 ffffffc0
[ 4757.943271] 3e10  fb163e40 ffffffc0 001ad398 ffffffc0 00000000 00000000 fb163ec0 ffffffc0
[ 4757.951578]
[ 4757.951578] X27: 0xffffffc000f3d050:
[ 4757.956623] d050  64656873 6e797320 75715f63 20657565 69646f6d 61636966 6e6f6974 0000000a
[ 4757.964925] d070  6d646326 6c3e2d61 006b636f 00000000 7376642f 7469672f 7269642f 672f7974
[ 4757.973231] d090  6d2d7469 65747361 696c5f72 2f78756e 6e72656b 642f6c65 65766972 762f7372
[ 4757.981535] d0b0  6f656469 6765742f 682f6172 2f74736f 6f68766e 635f7473 2e616d64 00000063
[ 4757.989846] d0d0  7376642f 7469672f 7269642f 672f7974 6d2d7469 65747361 696c5f72 2f78756e
[ 4757.998159] d0f0  6e72656b 642f6c65 65766972 762f7372 6f656469 6765742f 682f6172 2f74736f
[ 4758.006465] d110  6f68766e 695f7473 2e72746e 00000063 73253401 6d45203a 20797470 74696157
[ 4758.014777] d130  000a7265 00000000 73253401 6843203a 656e6e61 6e75206c 70616d2d 0a646570
[ 4758.023085]
[ 4758.023085] X28: 0xffffffc0f200bca8:
[ 4758.028129] bca8  00000000 00000000 f200bd50 ffffffc0 003736f4 ffffffc0 feec0320 ffffffc0
[ 4758.036435] bcc8  fee9d210 ffffffc0 00000004 00000000 3feb9500 ffffffc0 3febfe00 ffffffc0
[ 4758.044748] bce8  00000000 00000000 fee9d0a0 ffffffc0 f200bcf8 ffffffc0 f200bcf8 ffffffc0
[ 4758.053055] bd08  f200bd08 ffffffc0 f200bd08 ffffffc0 f200bd18 ffffffc0 f200bd18 ffffffc0
[ 4758.061359] bd28  f200bcf8 ffffffc0 f200bd08 ffffffc0 f200bd18 ffffffc0 00000000 00000000
[ 4758.069668] bd48  00000000 00000000 f200bd80 ffffffc0 00391f00 ffffffc0 feec0378 ffffffc0
[ 4758.077980] bd68  f9dd2d00 ffffffc0 00000000 00000000 f9dd2d00 ffffffc0 f200bd90 ffffffc0
[ 4758.086283] bd88  000c8158 ffffffc0 f200bde0 ffffffc0 000c8ad4 ffffffc0 f9dd2d00 ffffffc0
[ 4758.094589]
[ 4758.094589] X29: 0xffffffc0f200bb20:
[ 4758.099633] bb20  fb163db0 ffffffc0 00000000 00000000 00000000 00000000 00000001 00000000
[ 4758.107939] bb40  00000001 00000000 0046b2e8 00000000 00000010 00000000 00f3d0d0 ffffffc0
[ 4758.116252] bb60  f200bd28 ffffffc0 f200bba0 ffffffc0 000d9398 ffffffc0 f200bba0 ffffffc0
[ 4758.124559] bb80  f427a8c0 ffffffc0 800001c5 00000000 00000000 00000000 00000001 00000000
[ 4758.132870] bba0  f200bbf0 ffffffc0 000dbc68 ffffffc0 fb163da8 ffffffc0 00000140 00000000
[ 4758.141177] bbc0  00000000 00000000 00000001 00000000 00000001 00000000 003732d0 ffffffc0
[ 4758.149485] bbe0  00373270 ffffffc0 f200bd18 ffffffc0 f200bc30 ffffffc0 00373324 ffffffc0
[ 4758.157793] bc00  fb163da8 ffffffc0 f200bd18 ffffffc0 003732d0 ffffffc0 f200bd18 ffffffc0
[ 4758.167593] Process kworker/0:1H (pid: 934, stack limit = 0xffffffc0f2008060)
[ 4758.174718] Stack: (0xffffffc0f200bba0 to 0xffffffc0f200c000)
[ 4758.180460] bba0: f200bbf0 ffffffc0 000dbc68 ffffffc0 fb163da8 ffffffc0 00000140 00000000
[ 4758.188624] bbc0: 00000000 00000000 00000001 00000000 00000001 00000000 003732d0 ffffffc0
[ 4758.196792] bbe0: 00373270 ffffffc0 f200bd18 ffffffc0 f200bc30 ffffffc0 00373324 ffffffc0
[ 4758.204957] bc00: fb163da8 ffffffc0 f200bd18 ffffffc0 003732d0 ffffffc0 f200bd18 ffffffc0
[ 4758.213124] bc20: 00000100 00000000 f200bd28 ffffffc0 f200bc50 ffffffc0 00373124 ffffffc0
[ 4758.221289] bc40: e4916240 ffffffc0 00373180 ffffffc0 f200bcb0 ffffffc0 00373648 ffffffc0
[ 4758.229456] bc60: feec0320 ffffffc0 00000001 00000000 00000000 00000000 feec0330 ffffffc0
[ 4758.237622] bc80: fee9d210 ffffffc0 0015cf1e 00000000 00000000 00000000 3feb9500 ffffffc0
[ 4758.245787] bca0: 00000000 00000000 00000000 00000000 f200bd50 ffffffc0 003736f4 ffffffc0
[ 4758.253951] bcc0: feec0320 ffffffc0 fee9d210 ffffffc0 00000004 00000000 3feb9500 ffffffc0
[ 4758.262117] bce0: 3febfe00 ffffffc0 00000000 00000000 fee9d0a0 ffffffc0 f200bcf8 ffffffc0
[ 4758.270283] bd00: f200bcf8 ffffffc0 f200bd08 ffffffc0 f200bd08 ffffffc0 f200bd18 ffffffc0
[ 4758.278449] bd20: f200bd18 ffffffc0 f200bcf8 ffffffc0 f200bd08 ffffffc0 f200bd18 ffffffc0
[ 4758.286614] bd40: 00000000 00000000 00000000 00000000 f200bd80 ffffffc0 00391f00 ffffffc0
[ 4758.294778] bd60: feec0378 ffffffc0 f9dd2d00 ffffffc0 00000000 00000000 f9dd2d00 ffffffc0
[ 4758.302944] bd80: f200bd90 ffffffc0 000c8158 ffffffc0 f200bde0 ffffffc0 000c8ad4 ffffffc0
[ 4758.311111] bda0: f9dd2d00 ffffffc0 3feb9500 ffffffc0 f9dd2d30 ffffffc0 f2008000 ffffffc0
[ 4758.319278] bdc0: 00000000 00000000 0126847a ffffffc0 00f044b0 ffffffc0 000c89fc ffffffc0
[ 4758.327443] bde0: f200be30 ffffffc0 000cfb88 ffffffc0 ffdcfca0 ffffffc0 012c5568 ffffffc0
[ 4758.335608] be00: 00f02148 ffffffc0 f9dd2d00 ffffffc0 000c8950 ffffffc0 00000000 00000000
[ 4758.343775] be20: 00000000 00000000 00000000 00000000 00000000 00000000 00085510 ffffffc0
[ 4758.351941] be40: 000cfac8 ffffffc0 ffdcfca0 ffffffc0 00000000 00000000 00000000 00000000
[ 4758.360109] be60: 00000000 00000000 000dc034 ffffffc0 f200beb0 ffffffc0 00000000 00000000
[ 4758.368275] be80: 3fec68c0 ffffffc0 f9dd2d00 ffffffc0 00000000 00000000 00000000 00000000
[ 4758.376440] bea0: f200bea0 ffffffc0 f200bea0 ffffffc0 00000000 00000000 00000000 ffffffc0
[ 4758.384605] bec0: f200bec0 ffffffc0 f200bec0 ffffffc0 00000000 00000000 00000000 00000000
[ 4758.392773] bee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.400937] bf00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.409102] bf20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.417266] bf40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.425431] bf60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.433597] bf80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.441763] bfa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.449928] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000005 00000000
[ 4758.458094] bfe0: 00000000 00000000 00000000 00000000 202c3433 30302e30 610a3b7d 762e6277
[ 4758.466260] Call trace:
[ 4758.468704] [<ffffffc0f427a8c0>] 0xffffffc0f427a8c0
[ 4758.473579] [<ffffffc0000dbc64>] __wake_up+0x40/0x64
[ 4758.478539] [<ffffffc000373320>] action_wakeup_interruptible+0x50/0x60
[ 4758.485059] [<ffffffc000373120>] run_handlers+0x98/0x130
[ 4758.490365] [<ffffffc000373644>] process_wait_list+0x138/0x158
[ 4758.496192] [<ffffffc0003736f0>] nvhost_syncpt_thresh_fn+0x6c/0x84
[ 4758.502367] [<ffffffc000391efc>] syncpt_thresh_cascade_fn+0xc/0x18
[ 4758.508541] [<ffffffc0000c8154>] process_one_work+0x26c/0x42c
[ 4758.514280] [<ffffffc0000c8ad0>] worker_thread+0x180/0x294
[ 4758.519762] [<ffffffc0000cfb84>] kthread+0xbc/0xc8
[ 4758.524549] Code: f427a8a8 ffffffc0 004668c8 ffffffc0 (fe438000)
[ 4758.531435] ---[ end trace 3019f20b5740094f ]---
[ 4758.538000] BUG: sleeping function called from invalid context at /dvs/git/dirty/git-master_linux/kernel/kernel/rwsem.c:20
[ 4758.549105] in_atomic(): 1, irqs_disabled(): 0, pid: 934, name: kworker/0:1H
[ 4758.556169] CPU: 0 PID: 934 Comm: kworker/0:1H Tainted: G      D      3.10.67-g3a5c467 #1
[ 4758.564379] Workqueue: host_syncpt syncpt_thresh_cascade_fn
[ 4758.569985] Call trace:
[ 4758.572433] [<ffffffc000089ca0>] dump_backtrace+0x0/0xf4
[ 4758.577765] [<ffffffc000089f98>] show_stack+0x10/0x1c
[ 4758.582827] [<ffffffc0002f82a0>] dump_stack+0x1c/0x28
[ 4758.587917] [<ffffffc0000dc1f8>] __might_sleep+0xf0/0xfc
[ 4758.593252] [<ffffffc000aa5bfc>] down_read+0x20/0x38
[ 4758.598253] [<ffffffc0000bc704>] exit_signals+0x18/0x110
[ 4758.603587] [<ffffffc0000ab2f8>] do_exit+0xe8/0x3e0
[ 4758.608500] [<ffffffc00008a084>] die+0xe0/0xf4
[ 4758.612954] [<ffffffc00008a0cc>] arm64_notify_die+0x34/0x40
[ 4758.618560] [<ffffffc0000811c4>] do_undefinstr+0x1c4/0x1d8
[ 4758.624052] Exception stack(0xffffffc0f200b9e0 to 0xffffffc0f200bb00)
[ 4758.630514] b9e0: fb163e68 ffffffc0 fb163db0 ffffffc0 f200bba0 ffffffc0 f427a8c0 ffffffc0
[ 4758.638723] ba00: 00000004 00000000 00030001 ffffffc0 f427a8c0 ffffffc0 00000047 00000000
[ 4758.646908] ba20: 0000200c ffffff80 01317638 ffffffc0 f200bab0 ffffffc0 000813c8 ffffffc0
[ 4758.655122] ba40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4758.663316] ba60: f200bbd0 ffffffc0 00084e04 ffffffc0 f200bbd0 ffffffc0 00084e1c ffffffc0
[ 4758.671511] ba80: fb163de8 ffffffc0 00000001 00000000 00000000 00000000 00000000 00000000
[ 4758.679718] baa0: 00000000 00000000 f427a8c0 ffffffc0 ec34c3c0 ffffffc0 32d481ea 00000000
[ 4758.687911] bac0: fcda0fe0 ffffffc0 f200bc20 ffffffc0 3edd4000 00000000 eeffed88 00000000
[ 4758.696079] bae0: 000000f0 00000000 000000b5 00000000 000000de 00000000 00000024 00000000
[ 4758.704279] [<ffffffc000084c98>] el1_undef+0x4/0x5c
[ 4758.709188] [<ffffffc0000dbc64>] __wake_up+0x40/0x64
[ 4758.714167] [<ffffffc000373320>] action_wakeup_interruptible+0x50/0x60
[ 4758.720702] [<ffffffc000373120>] run_handlers+0x98/0x130
[ 4758.726009] [<ffffffc000373644>] process_wait_list+0x138/0x158
[ 4758.731850] [<ffffffc0003736f0>] nvhost_syncpt_thresh_fn+0x6c/0x84
[ 4758.738039] [<ffffffc000391efc>] syncpt_thresh_cascade_fn+0xc/0x18
[ 4758.744232] [<ffffffc0000c8154>] process_one_work+0x26c/0x42c
[ 4758.749988] [<ffffffc0000c8ad0>] worker_thread+0x180/0x294
[ 4758.755468] [<ffffffc0000cfb84>] kthread+0xbc/0xc8
[ 4758.760268] note: kworker/0:1H[934] exited with preempt_count 1
[ 4758.766323] Unable to handle kernel paging request at virtual address ffffffffffffffd8
[ 4758.774232] pgd = ffffffc0e6c90000
[ 4758.777632] [ffffffffffffffd8] *pgd=0000000000000000
[ 4758.782607] Internal error: Oops: 96000005 [#2] PREEMPT SMP
[ 4758.788171] Enter nvdumper_crash_setup_regs
[ 4758.792356] nvdumper: all registers are saved.
[ 4758.792361] nvdumper: all registers are saved.
[ 4758.792362] nvdumper: all registers are saved.
[ 4758.805670] nvdumper: all registers are saved.
[ 4758.810109] Modules linked in: rfcomm bnep bcmdhd cfg80211 nvhost_vi bluedroid_pm
[ 4758.817701] CPU: 0 PID: 934 Comm: kworker/0:1H Tainted: G      D      3.10.67-g3a5c467 #1
[ 4758.825873] task: ffffffc0fcda0ac0 ti: ffffffc0f2008000 task.ti: ffffffc0f2008000
[ 4758.833347] PC is at kthread_data+0x4/0xc
[ 4758.837354] LR is at wq_worker_sleeping+0x14/0xd4
[ 4758.842051] pc : [<ffffffc0000cff14>] lr : [<ffffffc0000c956c>] pstate: 800001c5
[ 4758.849434] sp : ffffffc0f200b7a0
[ 4758.852746] x29: ffffffc0f200b7a0 x28: ffffffc03feb98c0
[ 4758.858082] x27: ffffffc00130f868 x26: ffffffc00130f848
[ 4758.863420] x25: ffffffc0010e58c0 x24: ffffffc0010e58c0
[ 4758.868757] x23: ffffffc0fcda0ec8 x22: ffffffc0010e58c0
[ 4758.874095] x21: 0000000000000000 x20: ffffffc0fcda0ac0
[ 4758.879428] x19: 0000000000000000 x18: 000000000000000e
[ 4758.884765] x17: 0000000000000001 x16: 0000000000000007
[ 4758.890103] x15: 000000000000000e x14: 0000000000000014
[ 4758.895438] x13: 000000000000001a x12: ffffffc0f200b890
[ 4758.900774] x11: 000000000000001f x10: 0000000000000008
[ 4758.906108] x9 : 00000000009cfc81 x8 : ffffffc0f9624170
[ 4758.911445] x7 : 0000000000000000 x6 : 0000000000000000
[ 4758.916782] x5 : 0000000000000007 x4 : 0000000000000000
[ 4758.922120] x3 : ffffffc0012c5d60 x2 : 000000000000000f
[ 4758.927456] x1 : 0000000000000000 x0 : 0000000000000000
[ 4758.932792]
[ 4758.932792] PC: 0xffffffc0000cfe94:
[ 4758.937751] fe94  f9002fb3 f90033b3 90000001 9118a021 f90037a1 9101e3a1 f9003fa1 f90043a1
[ 4758.946056] feb4  910263a0 f0007181 91052021 d0008fa2 9115a042 940000aa 9100e274 aa1503e0
[ 4758.954362] fed4  aa1303e1 97ffffcc aa1403e0 9427588e a94153f3 f94013f5 a8cb7bfd d65f03c0
[ 4758.962671] fef4  910003e0 9272c400 f9400800 f941d800 f85c8000 d3420800 d65f03c0 f941d800
[ 4758.970982] ff14  f85d8000 d65f03c0 a9be7bfd 910003fd f941d801 910083a0 f81f8c1f d100a021
[ 4758.979287] ff34  d2800102 94020b4d f9400fa0 a8c27bfd d65f03c0 a9bf7bfd 910003fd 910003e0
[ 4758.987591] ff54  9272c400 f9400800 f941d800 d100e000 97fffdb7 a8c17bfd d65f03c0 52800000
[ 4758.995896] ff74  d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 97fffd85 aa0003e1 b4000060
[ 4759.004198]
[ 4759.004198] LR: 0xffffffc0000c94ec:
[ 4759.009156] 94ec  b9405802 52803900 6a00005f 540002a1 f9402660 b9400400 6b14001f 54000160
[ 4759.017463] 950c  f0008ce0 39527c00 35000100 f00071c0 9112c000 528064c1 97ff7016 52800021
[ 4759.025770] 952c  f0008ce0 39127c01 f9402660 910e0000 885f7c01 11000421 88027c01 35ffffa2
[ 4759.034077] 954c  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 2a0103f3 94001a6a
[ 4759.042387] 956c  b9405803 52803902 6a02007f 540004e1 f9402402 910003e0 9272c400 b9405400
[ 4759.050693] 958c  6b13001f 540001c0 f0008ce0 39528001 d2800000 35000401 f00071c0 9112c000
[ 4759.059002] 95ac  52806941 97ff6ff3 52800021 f0008ce0 39128001 d2800000 14000017 910e0043
[ 4759.067309] 95cc  885f7c61 51000421 8800fc61 35ffffa0 d5033bbf d2800000 350001e1 91006040
[ 4759.075613]
[ 4759.075613] SP: 0xffffffc0f200b720:
[ 4759.080572] b720  fcda0ac0 ffffffc0 00000000 00000000 010e58c0 ffffffc0 fcda0ec8 ffffffc0
[ 4759.088875] b740  010e58c0 ffffffc0 010e58c0 ffffffc0 0130f848 ffffffc0 0130f868 ffffffc0
[ 4759.097186] b760  3feb98c0 ffffffc0 f200b7a0 ffffffc0 000c956c ffffffc0 f200b7a0 ffffffc0
[ 4759.105491] b780  000cff14 ffffffc0 800001c5 00000000 3feb98c0 ffffffc0 fcda0ac0 ffffffc0
[ 4759.113798] b7a0  f200b7c0 ffffffc0 00aa6434 ffffffc0 3feb98c0 ffffffc0 00aa641c ffffffc0
[ 4759.122106] b7c0  f200b950 ffffffc0 00aa6950 ffffffc0 fcda0ac0 ffffffc0 0000000b 00000000
[ 4759.130411] b7e0  00000001 00000000 00efdd38 ffffffc0 800001c5 00000000 00000000 00000000
[ 4759.138714] b800  0046b2e8 00000000 00000010 00000000 00f3d0d0 ffffffc0 f200bd28 ffffffc0
[ 4759.147024] X3: 0xffffffc0012c5ce0:
[ 4759.151983] 5ce0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.160290] 5d00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.168599] 5d20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.176904] 5d40  ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000000
[ 4759.185208] 5d60  012ea0d0 ffffffc0 00000001 00000000 00000003 00000000 00000000 00000000
[ 4759.193517] 5d80  ffffffe0 0000000f 012c5d88 ffffffc0 012c5d88 ffffffc0 00112580 ffffffc0
[ 4759.201823] 5da0  ffc03600 ffffffc0 ffc03620 ffffffc0 00000400 00000000 00000000 00000000
[ 4759.210133] 5dc0  00000826 00000000 00000831 00000000 ffc03640 ffffffc0 ffc03660 ffffffc0
[ 4759.218440]
[ 4759.218440] X8: 0xffffffc0f96240f0:
[ 4759.223398] 40f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.231699] 4110  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.240008] 4130  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.248316] 4150  00000000 00000000 00000000 00000000 f9624160 ffffffc0 f9624160 ffffffc0
[ 4759.256625] 4170  f9624170 ffffffc0 f9624170 ffffffc0 f9624180 ffffffc0 f9624180 ffffffc0
[ 4759.264928] 4190  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.273238] 41b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.281546] 41d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.289858]
[ 4759.289858] X12: 0xffffffc0f200b810:
[ 4759.294904] b810  00f3d0d0 ffffffc0 f200bd28 ffffffc0 f200b890 ffffffc0 000a356c ffffffc0
[ 4759.303207] b830  fa874a40 ffffffc0 fcda0ac0 ffffffc0 f200b860 ffffffc0 00121e34 ffffffc0
[ 4759.311515] b850  00000000 00000000 0111b3c0 ffffffc0 f200b880 ffffffc0 001230d4 ffffffc0
[ 4759.319822] b870  f200b8b0 ffffffc0 00123320 ffffffc0 010e2058 ffffffc0 fcda14f8 ffffffc0
[ 4759.328121] b890  00000140 00000000 0111b3c0 ffffffc0 00000000 00000000 ffffffff 00000000
[ 4759.336427] b8b0  f200b8f0 ffffffc0 001233a4 ffffffc0 fcda0ac0 ffffffc0 01103040 ffffffc0
[ 4759.344734] b8d0  fcda0ac0 ffffffc0 000a9b40 ffffffc0 00000020 00000000 00000000 00000000
[ 4759.353036] b8f0  f200b900 ffffffc0 000aaec4 ffffffc0 f200b940 ffffffc0 000ab1a0 ffffffc0
[ 4759.361344]
[ 4759.361344] X20: 0xffffffc0fcda0a40:
[ 4759.366389] 0a40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.374696] 0a60  00000000 00000020 00000000 00000000 0000c350 00000000 0000c350 00000000
[ 4759.382999] 0a80  00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000
[ 4759.391303] 0aa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.399612] 0ac0  00000040 00000000 f2008000 ffffffc0 00000002 0420806c 00000000 00000000
[ 4759.407922] 0ae0  00000000 00000000 00000001 00000000 00000064 00000064 00000064 00000000
[ 4759.416233] 0b00  00ab07c8 ffffffc0 00015ab9 00000000 0000bd04 00000000 00000001 00000000
[ 4759.424538] 0b20  00000000 00000000 00000000 00000000 fcda0b30 ffffffc0 fcda0b30 ffffffc0
[ 4759.432844]
[ 4759.432844] X22: 0xffffffc0010e5840:
[ 4759.437890] 5840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.446187] 5860  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.454495] 5880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.462799] 58a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.471108] 58c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.479414] 58e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.487717] 5900  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.496026] 5920  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.504333]
[ 4759.504333] X23: 0xffffffc0fcda0e48:
[ 4759.509378] 0e48  00000000 00000000 fcda0e50 ffffffc0 00200200 00000000 f9624010 ffffffc0
[ 4759.517680] 0e68  00200200 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.525984] 0e88  00000000 00000000 00000650 00000000 00000000 00000000 00000650 00000000
[ 4759.534292] 0ea8  00000000 00000000 00000000 00000000 00000000 00000000 000ac313 00000000
[ 4759.542593] 0ec8  00000f10 00000000 0000000c 00000000 329dcd3f 00000000 0000000c 00000000
[ 4759.550900] 0ee8  329dcd3f 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.559206] 0f08  00000000 00000000 00000000 00000000 fcda0f18 ffffffc0 fcda0f18 ffffffc0
[ 4759.567514] 0f28  fcda0f28 ffffffc0 fcda0f28 ffffffc0 fcda0f38 ffffffc0 fcda0f38 ffffffc0
[ 4759.575822] X24: 0xffffffc0010e5840:
[ 4759.580866] 5840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.589176] 5860  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.597484] 5880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.605789] 58a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.614097] 58c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.622404] 58e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.630707] 5900  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.639008] 5920  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.647318]
[ 4759.647318] X25: 0xffffffc0010e5840:
[ 4759.652362] 5840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.660669] 5860  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.668976] 5880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.677284] 58a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.685592] 58c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.693895] 58e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.702204] 5900  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.710505] 5920  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.718808]
[ 4759.718808] X26: 0xffffffc00130f7c8:
[ 4759.723854] f7c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.732162] f7e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.740468] f808  00000000 00000000 00000000 00000000 00142064 ffffffc0 01268338 ffffffc0
[ 4759.748777] f828  00000000 00000000 00000000 00000000 01317668 ffffffc0 00000000 00000000
[ 4759.757082] f848  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.765387] f868  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.773692] f888  00000000 00000000 00000000 00000000 010e8580 ffffffc0 00000000 00000000
[ 4759.781994] f8a8  00000000 00000000 0130f8a8 ffffffc0 00000000 00000000 0130f8b8 ffffffc0
[ 4759.790308]
[ 4759.790308] X27: 0xffffffc00130f7e8:
[ 4759.795354] f7e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.803663] f808  00000000 00000000 00000000 00000000 00142064 ffffffc0 01268338 ffffffc0
[ 4759.811970] f828  00000000 00000000 00000000 00000000 01317668 ffffffc0 00000000 00000000
[ 4759.820270] f848  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.828574] f868  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.836886] f888  00000000 00000000 00000000 00000000 010e8580 ffffffc0 00000000 00000000
[ 4759.845192] f8a8  00000000 00000000 0130f8a8 ffffffc0 00000000 00000000 0130f8b8 ffffffc0
[ 4759.853498] f8c8  00000000 00000000 0130f8c8 ffffffc0 00000000 00000000 0130f8d8 ffffffc0
[ 4759.861808]
[ 4759.861808] X28: 0xffffffc03feb9840:
[ 4759.866852] 9840  00000000 00000000 00000001 00000000 00000000 00000000 ffc1e740 ffffffc0
[ 4759.875160] 9860  00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000
[ 4759.883469] 9880  00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.891779] 98a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4759.900081] 98c0  af31af30 00000000 000162b9 00000000 000162b9 00000000 00016260 00000000
[ 4759.908387] 98e0  000154ce 00000000 00011b09 00000000 0006cdaf 00000001 00000000 00000000
[ 4759.916691] 9900  00000000 00000000 00000000 00000000 fc936cd9 00000453 e352e000 0014464d
[ 4759.924995] 9920  028b0694 00000000 00000000 00000000 00000000 00000000 00073d71 00000000
[ 4759.933305] X29: 0xffffffc0f200b720:
[ 4759.938349] b720  fcda0ac0 ffffffc0 00000000 00000000 010e58c0 ffffffc0 fcda0ec8 ffffffc0
[ 4759.946651] b740  010e58c0 ffffffc0 010e58c0 ffffffc0 0130f848 ffffffc0 0130f868 ffffffc0
[ 4759.954961] b760  3feb98c0 ffffffc0 f200b7a0 ffffffc0 000c956c ffffffc0 f200b7a0 ffffffc0
[ 4759.963269] b780  000cff14 ffffffc0 800001c5 00000000 3feb98c0 ffffffc0 fcda0ac0 ffffffc0
[ 4759.971578] b7a0  f200b7c0 ffffffc0 00aa6434 ffffffc0 3feb98c0 ffffffc0 00aa641c ffffffc0
[ 4759.979881] b7c0  f200b950 ffffffc0 00aa6950 ffffffc0 fcda0ac0 ffffffc0 0000000b 00000000
[ 4759.988184] b7e0  00000001 00000000 00efdd38 ffffffc0 800001c5 00000000 00000000 00000000
[ 4759.996491] b800  0046b2e8 00000000 00000010 00000000 00f3d0d0 ffffffc0 f200bd28 ffffffc0
[ 4760.004798]
[ 4760.006289] Process kworker/0:1H (pid: 934, stack limit = 0xffffffc0f2008060)
[ 4760.013416] Stack: (0xffffffc0f200b7a0 to 0xffffffc0f200c000)
[ 4760.019157] b7a0: f200b7c0 ffffffc0 00aa6434 ffffffc0 3feb98c0 ffffffc0 00aa641c ffffffc0
[ 4760.027323] b7c0: f200b950 ffffffc0 00aa6950 ffffffc0 fcda0ac0 ffffffc0 0000000b 00000000
[ 4760.035489] b7e0: 00000001 00000000 00efdd38 ffffffc0 800001c5 00000000 00000000 00000000
[ 4760.043655] b800: 0046b2e8 00000000 00000010 00000000 00f3d0d0 ffffffc0 f200bd28 ffffffc0
[ 4760.051823] b820: f200b890 ffffffc0 000a356c ffffffc0 fa874a40 ffffffc0 fcda0ac0 ffffffc0
[ 4760.059989] b840: f200b860 ffffffc0 00121e34 ffffffc0 00000000 00000000 0111b3c0 ffffffc0
[ 4760.068154] b860: f200b880 ffffffc0 001230d4 ffffffc0 f200b8b0 ffffffc0 00123320 ffffffc0
[ 4760.076319] b880: 010e2058 ffffffc0 fcda14f8 ffffffc0 00000140 00000000 0111b3c0 ffffffc0
[ 4760.084487] b8a0: 00000000 00000000 ffffffff 00000000 f200b8f0 ffffffc0 001233a4 ffffffc0
[ 4760.092652] b8c0: fcda0ac0 ffffffc0 01103040 ffffffc0 fcda0ac0 ffffffc0 000a9b40 ffffffc0
[ 4760.100819] b8e0: 00000020 00000000 00000000 00000000 f200b900 ffffffc0 000aaec4 ffffffc0
[ 4760.108983] b900: f200b940 ffffffc0 000ab1a0 ffffffc0 fcda0ac0 ffffffc0 00000001 00000000
[ 4760.117149] b920: 00000001 00000000 00efdd38 ffffffc0 800001c5 00000000 000ab1b0 ffffffc0
[ 4760.125314] b940: f200b960 ffffffc0 000ab538 ffffffc0 f200b960 ffffffc0 000ab5ac ffffffc0
[ 4760.133481] b960: f200b990 ffffffc0 0008a088 ffffffc0 00000001 00000000 f2008000 ffffffc0
[ 4760.141647] b980: 00000000 00000000 0008a03c ffffffc0 f200b9c0 ffffffc0 0008a0d0 ffffffc0
[ 4760.149815] b9a0: f200ba80 ffffffc0 f427a8c0 ffffffc0 fe438000 00000000 f427a8c0 ffffffc0
[ 4760.157980] b9c0: f200b9d0 ffffffc0 000811c8 ffffffc0 f200bba0 ffffffc0 00084c9c ffffffc0
[ 4760.166145] b9e0: fb163e68 ffffffc0 fb163db0 ffffffc0 f200bba0 ffffffc0 f427a8c0 ffffffc0
[ 4760.174311] ba00: 00000004 00000000 00030001 ffffffc0 f427a8c0 ffffffc0 00000047 00000000
[ 4760.182479] ba20: 0000200c ffffff80 01317638 ffffffc0 f200bab0 ffffffc0 000813c8 ffffffc0
[ 4760.190646] ba40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.198813] ba60: f200bbd0 ffffffc0 00084e04 ffffffc0 f200bbd0 ffffffc0 00084e1c ffffffc0
[ 4760.206980] ba80: fb163de8 ffffffc0 00000001 00000000 00000000 00000000 00000000 00000000
[ 4760.215145] baa0: 00000000 00000000 f427a8c0 ffffffc0 ec34c3c0 ffffffc0 32d481ea 00000000
[ 4760.223311] bac0: fcda0fe0 ffffffc0 f200bc20 ffffffc0 3edd4000 00000000 eeffed88 00000000
[ 4760.231476] bae0: 000000f0 00000000 000000b5 00000000 000000de 00000000 00000024 00000000
[ 4760.239642] bb00: 00000000 00000000 00000000 00000000 00000000 00000000 fb163e68 ffffffc0
[ 4760.247807] bb20: fb163db0 ffffffc0 00000000 00000000 00000000 00000000 00000001 00000000
[ 4760.255975] bb40: 00000001 00000000 0046b2e8 00000000 00000010 00000000 00f3d0d0 ffffffc0
[ 4760.264140] bb60: f200bd28 ffffffc0 f200bba0 ffffffc0 000d9398 ffffffc0 f200bba0 ffffffc0
[ 4760.272306] bb80: f427a8c0 ffffffc0 800001c5 00000000 00000000 00000000 00000001 00000000
[ 4760.280471] bba0: f200bbf0 ffffffc0 000dbc68 ffffffc0 fb163da8 ffffffc0 00000140 00000000
[ 4760.288636] bbc0: 00000000 00000000 00000001 00000000 00000001 00000000 003732d0 ffffffc0
[ 4760.296802] bbe0: 00373270 ffffffc0 f200bd18 ffffffc0 f200bc30 ffffffc0 00373324 ffffffc0
[ 4760.304967] bc00: fb163da8 ffffffc0 f200bd18 ffffffc0 003732d0 ffffffc0 f200bd18 ffffffc0
[ 4760.313131] bc20: 00000100 00000000 f200bd28 ffffffc0 f200bc50 ffffffc0 00373124 ffffffc0
[ 4760.321298] bc40: e4916240 ffffffc0 00373180 ffffffc0 f200bcb0 ffffffc0 00373648 ffffffc0
[ 4760.329465] bc60: feec0320 ffffffc0 00000001 00000000 00000000 00000000 feec0330 ffffffc0
[ 4760.337630] bc80: fee9d210 ffffffc0 0015cf1e 00000000 00000000 00000000 3feb9500 ffffffc0
[ 4760.345798] bca0: 00000000 00000000 00000000 00000000 f200bd50 ffffffc0 003736f4 ffffffc0
[ 4760.353962] bcc0: feec0320 ffffffc0 fee9d210 ffffffc0 00000004 00000000 3feb9500 ffffffc0
[ 4760.362128] bce0: 3febfe00 ffffffc0 00000000 00000000 fee9d0a0 ffffffc0 f200bcf8 ffffffc0
[ 4760.370294] bd00: f200bcf8 ffffffc0 f200bd08 ffffffc0 f200bd08 ffffffc0 f200bd18 ffffffc0
[ 4760.378460] bd20: f200bd18 ffffffc0 f200bcf8 ffffffc0 f200bd08 ffffffc0 f200bd18 ffffffc0
[ 4760.386628] bd40: 00000000 00000000 00000000 00000000 f200bd80 ffffffc0 00391f00 ffffffc0
[ 4760.394793] bd60: feec0378 ffffffc0 f9dd2d00 ffffffc0 00000000 00000000 f9dd2d00 ffffffc0
[ 4760.402959] bd80: f200bd90 ffffffc0 000c8158 ffffffc0 f200bde0 ffffffc0 000c8ad4 ffffffc0
[ 4760.411126] bda0: f9dd2d00 ffffffc0 3feb9500 ffffffc0 f9dd2d30 ffffffc0 f2008000 ffffffc0
[ 4760.419291] bdc0: 00000000 00000000 0126847a ffffffc0 00f044b0 ffffffc0 000c89fc ffffffc0
[ 4760.427458] bde0: f200be30 ffffffc0 000cfb88 ffffffc0 ffdcfca0 ffffffc0 012c5568 ffffffc0
[ 4760.435623] be00: 00f02148 ffffffc0 f9dd2d00 ffffffc0 000c8950 ffffffc0 00000000 00000000
[ 4760.443789] be20: 00000000 00000000 00000000 00000000 00000000 00000000 00085510 ffffffc0
[ 4760.451953] be40: 000cfac8 ffffffc0 ffdcfca0 ffffffc0 00000000 00000000 00000000 00000000
[ 4760.460119] be60: 00000000 00000000 000dc034 ffffffc0 f200beb0 ffffffc0 00000000 00000000
[ 4760.468285] be80: 3fec68c0 ffffffc0 f9dd2d00 ffffffc0 00000000 00000000 00000000 00000000
[ 4760.476450] bea0: f200bea0 ffffffc0 f200bea0 ffffffc0 00000001 00000000 00010001 ffffffc0
[ 4760.484617] bec0: f200bec0 ffffffc0 f200bec0 ffffffc0 00000000 00000000 00000000 00000000
[ 4760.492782] bee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.500947] bf00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.509112] bf20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.517278] bf40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.525443] bf60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.533608] bf80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.541774] bfa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.549939] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000005 00000000
[ 4760.558104] bfe0: 00000000 00000000 00000000 00000000 202c3433 30302e30 610a3b7d 762e6277
[ 4760.566268] Call trace:
[ 4760.568716] [<ffffffc0000cff14>] kthread_data+0x4/0xc
[ 4760.573764] [<ffffffc000aa6430>] __schedule+0x12c/0x5fc
[ 4760.578984] [<ffffffc000aa694c>] schedule+0x4c/0x68
[ 4760.583857] [<ffffffc0000ab5a8>] do_exit+0x398/0x3e0
[ 4760.588814] [<ffffffc00008a084>] die+0xe0/0xf4
[ 4760.593254] [<ffffffc00008a0cc>] arm64_notify_die+0x34/0x40
[ 4760.598821] [<ffffffc0000811c4>] do_undefinstr+0x1c4/0x1d8
[ 4760.604301] Exception stack(0xffffffc0f200b9e0 to 0xffffffc0f200bb00)
[ 4760.610735] b9e0: fb163e68 ffffffc0 fb163db0 ffffffc0 f200bba0 ffffffc0 f427a8c0 ffffffc0
[ 4760.618901] ba00: 00000004 00000000 00030001 ffffffc0 f427a8c0 ffffffc0 00000047 00000000
[ 4760.627068] ba20: 0000200c ffffff80 01317638 ffffffc0 f200bab0 ffffffc0 000813c8 ffffffc0
[ 4760.635234] ba40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4760.643399] ba60: f200bbd0 ffffffc0 00084e04 ffffffc0 f200bbd0 ffffffc0 00084e1c ffffffc0
[ 4760.651566] ba80: fb163de8 ffffffc0 00000001 00000000 00000000 00000000 00000000 00000000
[ 4760.659731] baa0: 00000000 00000000 f427a8c0 ffffffc0 ec34c3c0 ffffffc0 32d481ea 00000000
[ 4760.667898] bac0: fcda0fe0 ffffffc0 f200bc20 ffffffc0 3edd4000 00000000 eeffed88 00000000
[ 4760.676063] bae0: 000000f0 00000000 000000b5 00000000 000000de 00000000 00000024 00000000
[ 4760.684228] [<ffffffc000084c98>] el1_undef+0x4/0x5c
[ 4760.689100] [<ffffffc0000dbc64>] __wake_up+0x40/0x64
[ 4760.694059] [<ffffffc000373320>] action_wakeup_interruptible+0x50/0x60
[ 4760.700577] [<ffffffc000373120>] run_handlers+0x98/0x130
[ 4760.705884] [<ffffffc000373644>] process_wait_list+0x138/0x158
[ 4760.711709] [<ffffffc0003736f0>] nvhost_syncpt_thresh_fn+0x6c/0x84
[ 4760.717883] [<ffffffc000391efc>] syncpt_thresh_cascade_fn+0xc/0x18
[ 4760.724057] [<ffffffc0000c8154>] process_one_work+0x26c/0x42c
[ 4760.729797] [<ffffffc0000c8ad0>] worker_thread+0x180/0x294
[ 4760.735277] [<ffffffc0000cfb84>] kthread+0xbc/0xc8
[ 4760.740062] Code: f85c8000 d3420800 d65f03c0 f941d800 (f85d8000)
[ 4788.227570] ------------[ cut here ]------------
[ 4788.232201] WARNING: at /dvs/git/dirty/git-master_linux/kernel/kernel/watchdog.c:269 watchdog_check_hardlockup_other_cpu+0x114/0x160()
[ 4788.244268] Watchdog detected hard LOCKUP on cpu 3
[ 4788.248880] Modules linked in: rfcomm bnep bcmdhd cfg80211 nvhost_vi bluedroid_pm
[ 4788.256658] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D      3.10.67-g3a5c467 #1
[ 4788.264390] Call trace:
[ 4788.266835] [<ffffffc000089ca0>] dump_backtrace+0x0/0xf4
[ 4788.272139] [<ffffffc000089f98>] show_stack+0x10/0x1c
[ 4788.277185] [<ffffffc0002f82a0>] dump_stack+0x1c/0x28
[ 4788.282231] [<ffffffc0000a54a4>] warn_slowpath_common+0x74/0x9c
[ 4788.288144] [<ffffffc0000a5518>] warn_slowpath_fmt+0x4c/0x58
[ 4788.293795] [<ffffffc000119c70>] watchdog_check_hardlockup_other_cpu+0x110/0x160
[ 4788.301183] [<ffffffc000119da4>] watchdog_timer_fn+0x50/0x1ec
[ 4788.306922] [<ffffffc0000d3554>] __run_hrtimer+0x164/0x27c
[ 4788.312402] [<ffffffc0000d3f0c>] hrtimer_interrupt+0x144/0x25c
[ 4788.318228] [<ffffffc000780008>] tegra210_timer_isr+0x38/0x4c
[ 4788.323968] [<ffffffc00011ad84>] handle_irq_event_percpu+0xcc/0x210
[ 4788.330227] [<ffffffc00011af0c>] handle_irq_event+0x44/0x74
[ 4788.335792] [<ffffffc00011e0e4>] handle_fasteoi_irq+0xc0/0x10c
[ 4788.341618] [<ffffffc00011a254>] generic_handle_irq+0x24/0x40
[ 4788.347357] [<ffffffc0000859f4>] handle_IRQ+0x8c/0xc4
[ 4788.352401] [<ffffffc0000813cc>] gic_handle_irq+0x70/0x184
[ 4788.357880] Exception stack(0xffffffc0fff6bd90 to 0xffffffc0fff6beb0)
[ 4788.364312] bd80:                                     3fed22f0 ffffffc0 00000002 00000000
[ 4788.372478] bda0: fff6bef0 ffffffc0 00743c2c ffffffc0 80000145 00000000 00000001 00000000
[ 4788.380643] bdc0: 8007b000 00000000 8007d000 00000000 00000000 00000000 00743c28 ffffffc0
[ 4788.388809] bde0: 3575ad75 05e59ebe 1c3c51c7 00000000 bd518400 00000454 34152af5 00000000
[ 4788.396973] be00: 3b9aca00 00000000 32d481ea 00000000 00000018 00000000 00000000 00000000
[ 4788.405141] be20: 0006cf90 00000001 ef0fed88 00000000 000000f0 00000000 0000008b 00000000
[ 4788.413308] be40: 00000020 00000000 00000015 00000000 00000000 00000000 00000000 00000000
[ 4788.421475] be60: 00000000 00000000 3fed22f0 ffffffc0 00000002 00000000 d5c4c41f 0000045a
[ 4788.429642] be80: d5b44cae 0000045a 3fed2608 ffffffc0 00000002 00000000 8007b000 00000000
[ 4788.437806] bea0: 8007d000 00000000 000801f0 ffffffc0
[ 4788.442852] [<ffffffc000084e00>] el1_irq+0x80/0xf0
[ 4788.447637] [<ffffffc000743e10>] cpuidle_idle_call+0x178/0x278
[ 4788.453463] [<ffffffc00008670c>] arch_cpu_idle+0x8/0x24
[ 4788.458682] [<ffffffc0000f57c4>] cpu_idle_loop+0x20c/0x278
[ 4788.464161] [<ffffffc0000f584c>] cpu_startup_entry+0x1c/0x20
[ 4788.469815] [<ffffffc000aa295c>] secondary_start_kernel+0x134/0x148
[ 4788.476075] ---[ end trace 3019f20b57400950 ]---

In this state, I can’t even use the serial port for entering commands, all it does is write those logs continuously.
The only customization I made on the boards: I removed Unity and replaced it with Openbox and installed the libs needed for WebRTC.

When WebRTC isn’t running, the board runs fine during multiple days without any issue.

Thanks for any help.

linuxdev · January 27, 2016, 4:26pm

The above of course looks like a memory error on eMMC, but I’m wondering if you have a regular SATA hard drive you might be able to copy over the rootfs to and add a new u-boot entry to boot to that instead (leaving the original in tact). The goal being to be able to boot to an identical rootfs running on a real non-solid-state drive and see if it changes (just remember to not copy pseudo file systems over, e.g., /proc and /sys…it wouldn’t hurt, but also would not help to copy over /boot).

Bibek · March 18, 2016, 4:46am

Dear Customer,

Please try with after applying this attached patch

regards
Bibek
video-tegra-host-use-dynamically-allocated-wait-queue.patch.tar (10 KB)

Charlinux · April 26, 2016, 3:42pm

Hi,

Sorry for my very late reply…
I managed to find some time to try the patch you gave but the problem is still the same.

This time, I got those logs on the serial port:

[ 8663.660185] BUG: soft lockup - CPU#2 stuck for 22s! [nvmap-bz:57]
[ 8691.660185] BUG: soft lockup - CPU#2 stuck for 22s! [nvmap-bz:57]
[ 8719.660185] BUG: soft lockup - CPU#2 stuck for 22s! [nvmap-bz:57]
[ 8747.660185] BUG: soft lockup - CPU#2 stuck for 22s! [nvmap-bz:57]
[ 8775.660186] BUG: soft lockup - CPU#2 stuck for 22s! [nvmap-bz:57]
[ 8803.660185] BUG: soft lockup - CPU#2 stuck for 23s! [nvmap-bz:57]
[ 8831.660185] BUG: soft lockup - CPU#2 stuck for 23s! [nvmap-bz:57]
[ 8859.660186] BUG: soft lockup - CPU#2 stuck for 23s! [nvmap-bz:57]

I’m not sure if this could be caused by an error in the software running on the board or if it’s a bug in the system…

Thanks for any help.

Bibek · April 29, 2016, 11:02am

Hi Charlinux,

Please share the full kernel bootlog till the issue is seen from a uart console.
Also after the boot,run the below command to increase the log level. so that we see the backtrace after softlockup

echo 9 > /proc/sysrq-trigger

regards
Bibek

Charlinux · May 3, 2016, 8:07am

Hi,

Thank you for your help.

Okay, so I’ve echoed 9 into sysrq-trigger then launched my app.
Here is the full kernel log when it froze:

Today, I got something new. The board froze when it was doing nothing. My app was not started, only X server, screensaver. The uart was connected but unfortunatly, I didn’t echo 9 this time…
Anyway, here are the logs:

[10088.640793] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[10088.648880] pgd = ffffffc00007d000
[10088.652278] [00000000] *pgd=000000017fc06003, *pmd=000000017fc07003, *pte=00e0000050041407
[10088.660571] Internal error: Oops: 96000007 [#1] PREEMPT SMP
[10088.666135] Enter nvdumper_crash_setup_regs
[10088.670360] nvdumper: all registers are saved.
[10088.670376] nvdumper: all registers are saved.
[10088.670383] nvdumper: all registers are saved.
[10088.683671] nvdumper: all registers are saved.
[10088.688110] Modules linked in: bcmdhd cfg80211 nvhost_vi bluedroid_pm
[10088.694618] CPU: 0 PID: 3188 Comm: kworker/0:1H Not tainted 3.10.67-g458d45c #1
[10088.701925] Workqueue: host_syncpt syncpt_thresh_cascade_fn
[10088.707505] task: ffffffc0f68435c0 ti: ffffffc0e0094000 task.ti: ffffffc0e0094000
[10088.714978] PC is at __wake_up_common+0x34/0xa4
[10088.719503] LR is at __wake_up+0x44/0x64
[10088.723421] pc : [<ffffffc0000d9378>] lr : [<ffffffc0000dbc78>] pstate: 800003c5
[10088.730804] sp : ffffffc0e0097ba0
[10088.734113] x29: ffffffc0e0097ba0 x28: ffffffc0e0097d28
[10088.739444] x27: ffffffc000f44a50 x26: 0000000000000010
[10088.744774] x25: ffffffc000373580 x24: 0000000000000001
[10088.750104] x23: 0000000000000001 x22: 0000000000000000
[10088.755433] x21: 0000000000000000 x20: ffffffc0f8f37db0
[10088.760765] x19: ffffffc0f8f37da8 x18: ca2485a68a59eb39
[10088.766095] x17: 191aa240a0c8c007 x16: 249a446970c6a070
[10088.771426] x15: 04b540a45a1ca5f1 x14: 8960600e45a1089a
[10088.776755] x13: e8c194726942d692 x12: 0000000000000000
[10088.782085] x11: 191818f62746b2c5 x10: 00000001000eefb2
[10088.787416] x9 : ffffffc0e0097c20 x8 : ffffffc0f6843ae0
[10088.792745] x7 : 000000001e094dc7 x6 : 0000000000000000
[10088.798074] x5 : ffffffc0feea0358 x4 : 0000000000000000
[10088.803403] x3 : 0000000000000000 x2 : 0000000000000001
[10088.808732] x1 : 0000000000000000 x0 : 0000000000000000
[10088.814062]
[10088.814062] PC: 0xffffffc0000d92f8:
[10088.819020] 92f8  eb04003f 54000222 7100045f 54000061 9ac12460 1400000d 90004ee0 912fc000
[10088.827306] 9318  91052000 8b22cc02 36000081 39400040 9b037c03 d347fc63 d341fc21 91000442
[10088.835594] 9338  b5ffff41 aa0303e0 d65f03c0 a9bb7bfd 910003fd a90153f3 a9025bf5 a90363f7
[10088.843876] 9358  f90023f9 2a0103f7 2a0203f8 2a0303f6 aa0403f5 aa0003f4 f8408e81 aa0103e0
[10088.852164] 9378  f85e8413 d1006273 eb01029f 54000061 14000012 aa0503f3 b9400019 f9400805
[10088.860449] 9398  2a1703e1 2a1603e2 aa1503e3 d63f00a0 34000080 36000079 71000718 540000e0
[10088.868734] 93b8  f9400e65 d10060a5 aa1303e0 91006273 eb13029f 54fffe01 a94153f3 a9425bf5
[10088.877019] 93d8  a94363f7 f94023f9 a8c57bfd d65f03c0 a9bf7bfd 910003fd 52800003 d2800004
[10088.885303]
[10088.885303] LR: 0xffffffc0000dbbf8:
[10088.890260] bbf8  b9400842 34000042 f820683f f9400a81 aa0103e0 90009242 91070042 f873d842
[10088.898543] bc18  f8626800 b9400000 34000040 f822683f a94153f3 a8c27bfd d65f03c0 a9bc7bfd
[10088.906826] bc38  910003fd a90153f3 a9025bf5 f9001bf7 aa0003f3 2a0103f7 2a0203f6 aa0303f5
[10088.915106] bc58  942749fa aa0003f4 aa1303e0 2a1703e1 2a1603e2 52800003 aa1503e4 97fff5b4
[10088.923385] bc78  aa1303e0 aa1403e1 94274a6c a94153f3 a9425bf5 f9401bf7 a8c47bfd d65f03c0
[10088.931664] bc98  a9bc7bfd 910003fd a90153f3 a9025bf5 f9001bf7 b4000220 aa0303f5 2a0203f4
[10088.939950] bcb8  2a0103f6 aa0003f3 942749e0 aa0003f7 6b1f029f aa1303e0 2a1603e1 2a1403e2
[10088.948237] bcd8  1a9f07e3 aa1503e4 97fff599 aa1303e0 aa1703e1 94274a51 a94153f3 a9425bf5
[10088.956521]
[10088.956521] SP: 0xffffffc0e0097b20:
[10088.961477] 7b20  f8f37db0 ffffffc0 00000000 00000000 00000000 00000000 00000001 00000000
[10088.969762] 7b40  00000001 00000000 00373580 ffffffc0 00000010 00000000 00f44a50 ffffffc0
[10088.978052] 7b60  e0097d28 ffffffc0 e0097ba0 ffffffc0 000dbc78 ffffffc0 e0097ba0 ffffffc0
[10088.986338] 7b80  000d9378 ffffffc0 800003c5 00000000 00000000 00000000 00000001 00000000
[10088.994621] 7ba0  e0097bf0 ffffffc0 000dbc78 ffffffc0 f8f37da8 ffffffc0 00000340 00000000
[10089.002901] 7bc0  00000000 00000000 00000001 00000000 00000001 00000000 003735e0 ffffffc0
[10089.011180] 7be0  00373580 ffffffc0 e0097d18 ffffffc0 e0097c30 ffffffc0 00373634 ffffffc0
[10089.019464] 7c00  f8f37da8 ffffffc0 e0097d18 ffffffc0 003735e0 ffffffc0 e0097d18 ffffffc0
[10089.027752]
[10089.027752] X5: 0xffffffc0feea02d8:
[10089.032709] 02d8  00000000 00000000 ffffffe0 0000000f feea02e8 ffffffc0 feea02e8 ffffffc0
[10089.040990] 02f8  003734c8 ffffffc0 feea0300 ffffffc0 feea0300 ffffffc0 feea0310 ffffffc0
[10089.049273] 0318  feea0310 ffffffc0 fed74a10 ffffffc0 00000004 00000000 00000001 11961196
[10089.057560] 0338  feea0338 ffffffc0 feea0338 ffffffc0 00000000 00000000 00000000 00000000
[10089.065844] 0358  e833eac0 ffffffc0 e833ef40 ffffffc0 74736f68 5f70735f 00003430 00000000
[10089.074124] 0378  00000020 00000000 feea0380 ffffffc0 feea0380 ffffffc0 00390ee4 ffffffc0
[10089.082409] 0398  00002768 00000000 2627297e 00000000 000000c0 00000000 feea03b0 ffffffc0
[10089.090693] 03b8  feea03b0 ffffffc0 003734c8 ffffffc0 feea03c8 ffffffc0 feea03c8 ffffffc0
[10089.098979]
[10089.098979] X8: 0xffffffc0f6843a60:
[10089.103935] 3a60  48313a30 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.112221] 3a80  fe4c4080 ffffffc0 f68435c0 ffffffc0 fe4c4080 ffffffc0 010ef8c0 ffffffc0
[10089.120499] 3aa0  00000000 00000000 010ef8c0 ffffffc0 010ef8c0 ffffffc0 01322e48 ffffffc0
[10089.128787] 3ac0  01322e68 ffffffc0 fdca8900 ffffffc0 e00978e0 ffffffc0 e00978e0 ffffffc0
[10089.137074] 3ae0  00086b14 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[10089.145358] 3b00  20801062 54b51873 f482b3f8 048c8a05 8293af38 b42a280e b3812191 226ae050
[10089.153642] 3b20  7980250c daa2b861 48f658a1 0c1310a7 bd5d004c d3a21846 6308880e 188484a2
[10089.161924] 3b40  74d07687 a2e142c8 4104a0d2 c394770d 4a3da5bc 0e81e983 0f5a205a 810d2c0f
[10089.170211]
[10089.170211] X9: 0xffffffc0e0097ba0:
[10089.175168] 7ba0  e0097bf0 ffffffc0 000dbc78 ffffffc0 f8f37da8 ffffffc0 00000340 00000000
[10089.183452] 7bc0  00000000 00000000 00000001 00000000 00000001 00000000 003735e0 ffffffc0
[10089.191739] 7be0  00373580 ffffffc0 e0097d18 ffffffc0 e0097c30 ffffffc0 00373634 ffffffc0
[10089.200021] 7c00  f8f37da8 ffffffc0 e0097d18 ffffffc0 003735e0 ffffffc0 e0097d18 ffffffc0
[10089.208304] 7c20  00000100 00000000 00000010 00000000 e0097c50 ffffffc0 00373434 ffffffc0
[10089.216588] 7c40  e833e940 ffffffc0 00aacd90 ffffffc0 e0097cb0 ffffffc0 00373958 ffffffc0
[10089.224878] 7c60  feea0320 ffffffc0 00000001 00000000 00000000 00000000 feea0330 ffffffc0
[10089.233161] 7c80  fed74a10 ffffffc0 004c6cf6 00000000 00000000 00000000 3feb7500 ffffffc0
[10089.241448]
[10089.241448] X19: 0xffffffc0f8f37d28:
[10089.246493] 7d28  0000008e 00000000 00000180 00000000 00000129 00000000 0008e000 ffffffc0
[10089.254780] 7d48  f8f34000 ffffffc0 00000000 00000000 004c6e4c ffffffc0 f8f37d80 ffffffc0
[10089.263067] 7d68  003709e0 ffffffc0 f8f37d90 ffffffc0 00000000 00000000 fcf18ac0 00000000
[10089.271349] 7d88  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.279632] 7da8  00000000 00000000 00000000 00000000 f8f37d90 ffffffc0 f8f37e00 ffffffc0
[10089.287910] 7dc8  000dec44 ffffffc0 ab039db8 00000000 00000010 00000000 00000000 00000000
[10089.296197] 7de8  fff8e0f8 ffffffc0 f8f37e20 ffffffc0 00000006 00000000 f8f37e80 00000000
[10089.304481] 7e08  001ad4b4 ffffffc0 f8f37e40 ffffffc0 001ad4c4 ffffffc0 00000000 00000000
[10089.312763]
[10089.312763] X20: 0xffffffc0f8f37d30:
[10089.317806] 7d30  fed74a10 00000000 00000129 00000000 0008e000 ffffffc0 f8f34000 ffffffc0
[10089.326094] 7d50  f8f37d90 00000000 004c6e78 ffffffc0 000001c0 00000000 fd586248 ffffffc0
[10089.334378] 7d70  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.342662] 7d90  00000000 00000000 00000000 ffffffc0 00000001 00000000 00aae6b0 ffffffc0
[10089.350941] 7db0  f8f37d90 ffffffc0 f8f37d90 ffffffc0 f8f37e00 ffffffc0 000dec44 ffffffc0
[10089.359222] 7dd0  c0104806 00000000 00000000 00000000 00000000 00000000 fff8e148 00000000
[10089.367503] 7df0  00000000 00000000 00000000 00000000 f8f37e40 ffffffc0 001ad4b4 ffffffc0
[10089.375783] 7e10  f8f37e40 ffffffc0 001ad4c4 00000080 00000001 00000000 fff8e0f8 00000000
[10089.384074]
[10089.384074] X25: 0xffffffc000373500:
[10089.389118] 3500  aa1303e3 91010294 aa0203e4 f9000042 f9000442 f9400425 f9400026 eb0100df
[10089.397405] 3520  54000140 eb05003f 54000100 f94000a7 f9000046 f90004c2 f9000445 f90000a2
[10089.405687] 3540  f9000027 f90004e1 f9000c64 91004042 91004021 91002063 eb14003f 54fffd61
[10089.413975] 3560  aa1503e0 941ce138 910163a0 97ffff8b a94153f3 f94013f5 a8c87bfd d65f03c0
[10089.422255] 3580  a9be7bfd 910003fd f9000bf3 f9401813 d2800061 91007000 885f7c02 8803fc01
[10089.430532] 35a0  35ffffc3 d5033bbf 7100045f 540000a0 b0005e80 91294000 52801821 97f4c7f2
[10089.438815] 35c0  aa1303e0 52800061 52800022 d2800003 97f5a199 f9400bf3 a8c27bfd d65f03c0
[10089.447102] 35e0  a9be7bfd 910003fd f9000bf3 f9401813 d2800061 91007000 885f7c02 8803fc01
[10089.455387]
[10089.455387] X27: 0xffffffc000f449d0:
[10089.460432] 49d0  64656873 6e797320 75715f63 20657565 69646f6d 61636966 6e6f6974 0000000a
[10089.468718] 49f0  6d646326 6c3e2d61 006b636f 00000000 7376642f 7469672f 7269642f 672f7974
[10089.477004] 4a10  6d2d7469 65747361 696c5f72 2f78756e 6e72656b 642f6c65 65766972 762f7372
[10089.485291] 4a30  6f656469 6765742f 682f6172 2f74736f 6f68766e 635f7473 2e616d64 00000063
[10089.493574] 4a50  7376642f 7469672f 7269642f 672f7974 6d2d7469 65747361 696c5f72 2f78756e
[10089.501855] 4a70  6e72656b 642f6c65 65766972 762f7372 6f656469 6765742f 682f6172 2f74736f
[10089.510138] 4a90  6f68766e 695f7473 2e72746e 00000063 73253401 6d45203a 20797470 74696157
[10089.518422] 4ab0  000a7265 00000000 73253401 6843203a 656e6e61 6e75206c 70616d2d 0a646570
[10089.526706]
[10089.526706] X28: 0xffffffc0e0097ca8:
[10089.531750] 7ca8  00000000 00000000 e0097d50 ffffffc0 00373a04 ffffffc0 feea0320 ffffffc0
[10089.540034] 7cc8  fed74a10 ffffffc0 00000004 00000000 3feb7500 ffffffc0 3febde00 ffffffc0
[10089.548317] 7ce8  00000000 00000000 fed748a0 ffffffc0 e0097cf8 ffffffc0 e0097cf8 ffffffc0
[10089.556603] 7d08  e0097d08 ffffffc0 e0097d08 ffffffc0 e0097d18 ffffffc0 e0097d18 ffffffc0
[10089.564887] 7d28  e0097cf8 ffffffc0 e0097d08 ffffffc0 e0097d18 ffffffc0 00000000 00000000
[10089.573171] 7d48  00000000 00000000 e0097d80 ffffffc0 00390ef4 ffffffc0 feea0378 ffffffc0
[10089.581453] 7d68  e56c9a80 ffffffc0 00000000 00000000 e56c9a80 ffffffc0 e0097d90 ffffffc0
[10089.589732] 7d88  000c8168 ffffffc0 e0097de0 ffffffc0 000c8ae4 ffffffc0 e56c9a80 ffffffc0
[10089.598017]
[10089.598017] X29: 0xffffffc0e0097b20:
[10089.603062] 7b20  f8f37db0 ffffffc0 00000000 00000000 00000000 00000000 00000001 00000000
[10089.611346] 7b40  00000001 00000000 00373580 ffffffc0 00000010 00000000 00f44a50 ffffffc0
[10089.619628] 7b60  e0097d28 ffffffc0 e0097ba0 ffffffc0 000dbc78 ffffffc0 e0097ba0 ffffffc0
[10089.627917] 7b80  000d9378 ffffffc0 800003c5 00000000 00000000 00000000 00000001 00000000
[10089.636197] 7ba0  e0097bf0 ffffffc0 000dbc78 ffffffc0 f8f37da8 ffffffc0 00000340 00000000
[10089.644482] 7bc0  00000000 00000000 00000001 00000000 00000001 00000000 003735e0 ffffffc0
[10089.652766] 7be0  00373580 ffffffc0 e0097d18 ffffffc0 e0097c30 ffffffc0 00373634 ffffffc0
[10089.661050] 7c00  f8f37da8 ffffffc0 e0097d18 ffffffc0 003735e0 ffffffc0 e0097d18 ffffffc0
[10089.669334]
[10089.670825] Process kworker/0:1H (pid: 3188, stack limit = 0xffffffc0e0094060)
[10089.678037] Stack: (0xffffffc0e0097ba0 to 0xffffffc0e0098000)
[10089.683777] 7ba0: e0097bf0 ffffffc0 000dbc78 ffffffc0 f8f37da8 ffffffc0 00000340 00000000
[10089.691945] 7bc0: 00000000 00000000 00000001 00000000 00000001 00000000 003735e0 ffffffc0
[10089.700111] 7be0: 00373580 ffffffc0 e0097d18 ffffffc0 e0097c30 ffffffc0 00373634 ffffffc0
[10089.708275] 7c00: f8f37da8 ffffffc0 e0097d18 ffffffc0 003735e0 ffffffc0 e0097d18 ffffffc0
[10089.716441] 7c20: 00000100 00000000 00000010 00000000 e0097c50 ffffffc0 00373434 ffffffc0
[10089.724607] 7c40: e833e940 ffffffc0 00aacd90 ffffffc0 e0097cb0 ffffffc0 00373958 ffffffc0
[10089.732773] 7c60: feea0320 ffffffc0 00000001 00000000 00000000 00000000 feea0330 ffffffc0
[10089.740938] 7c80: fed74a10 ffffffc0 004c6cf6 00000000 00000000 00000000 3feb7500 ffffffc0
[10089.749103] 7ca0: 00000000 00000000 00000000 00000000 e0097d50 ffffffc0 00373a04 ffffffc0
[10089.757270] 7cc0: feea0320 ffffffc0 fed74a10 ffffffc0 00000004 00000000 3feb7500 ffffffc0
[10089.765436] 7ce0: 3febde00 ffffffc0 00000000 00000000 fed748a0 ffffffc0 e0097cf8 ffffffc0
[10089.773603] 7d00: e0097cf8 ffffffc0 e0097d08 ffffffc0 e0097d08 ffffffc0 e0097d18 ffffffc0
[10089.781770] 7d20: e0097d18 ffffffc0 e0097cf8 ffffffc0 e0097d08 ffffffc0 e0097d18 ffffffc0
[10089.789935] 7d40: 00000000 00000000 00000000 00000000 e0097d80 ffffffc0 00390ef4 ffffffc0
[10089.798101] 7d60: feea0378 ffffffc0 e56c9a80 ffffffc0 00000000 00000000 e56c9a80 ffffffc0
[10089.806267] 7d80: e0097d90 ffffffc0 000c8168 ffffffc0 e0097de0 ffffffc0 000c8ae4 ffffffc0
[10089.814432] 7da0: e56c9a80 ffffffc0 3feb7500 ffffffc0 e56c9ab0 ffffffc0 e0094000 ffffffc0
[10089.822598] 7dc0: 00000000 00000000 0127b47a ffffffc0 00f0bdb0 ffffffc0 000c8a0c ffffffc0
[10089.830763] 7de0: e0097e30 ffffffc0 000cfb98 ffffffc0 ffdcfca0 ffffffc0 012d8b68 ffffffc0
[10089.838928] 7e00: 00f09a48 ffffffc0 e56c9a80 ffffffc0 000c8960 ffffffc0 00000000 00000000
[10089.847094] 7e20: 00000000 00000000 00000000 00000000 00000000 00000000 00085510 ffffffc0
[10089.855259] 7e40: 000cfad8 ffffffc0 ffdcfca0 ffffffc0 00000000 00000000 00000000 00000000
[10089.863426] 7e60: 00000000 00000000 000dc044 ffffffc0 e0097eb0 ffffffc0 00000000 00000000
[10089.871592] 7e80: 3fede8c0 ffffffc0 e56c9a80 ffffffc0 00000000 00000000 00000000 00000000
[10089.879759] 7ea0: e0097ea0 ffffffc0 e0097ea0 ffffffc0 00000000 00000000 00000000 ffffffc0
[10089.887924] 7ec0: e0097ec0 ffffffc0 e0097ec0 ffffffc0 00000000 00000000 00000000 00000000
[10089.896089] 7ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.904255] 7f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.912419] 7f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.920586] 7f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.928750] 7f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.936915] 7f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.945082] 7fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[10089.953246] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000005 00000000
[10089.961411] 7fe0: 00000000 00000000 00000000 00000000 00040bb7 000386b4 00000001 00000000
[10089.969574] Call trace:
[10089.972023] [<ffffffc0000d9378>] __wake_up_common+0x34/0xa4
[10089.977587] [<ffffffc0000dbc74>] __wake_up+0x40/0x64
[10089.982547] [<ffffffc000373630>] action_wakeup_interruptible+0x50/0x60
[10089.989067] [<ffffffc000373430>] run_handlers+0x98/0x130
[10089.994372] [<ffffffc000373954>] process_wait_list+0x138/0x158
[10090.000199] [<ffffffc000373a00>] nvhost_syncpt_thresh_fn+0x6c/0x84
[10090.006371] [<ffffffc000390ef0>] syncpt_thresh_cascade_fn+0xc/0x18
[10090.012544] [<ffffffc0000c8164>] process_one_work+0x26c/0x42c
[10090.018284] [<ffffffc0000c8ae0>] worker_thread+0x180/0x294
[10090.023763] [<ffffffc0000cfb94>] kthread+0xbc/0xc8
[10090.028549] Code: aa0403f5 aa0003f4 f8408e81 aa0103e0 (f85e8413)
[10090.034790] ---[ end trace 740f779013aef9b3 ]---
[10090.042606] note: kworker/0:1H[3188] exited with preempt_count 1
[10090.048663] Unable to handle kernel paging request at virtual address ffffffffffffffd8
[10090.056569] pgd = ffffffc0e925f000
[10090.059967] [ffffffffffffffd8] *pgd=0000000000000000
[10090.064942] Internal error: Oops: 96000005 [#2] PREEMPT SMP
[10090.070505] Enter nvdumper_crash_setup_regs

Regards,
Charlinux

linuxdev · May 3, 2016, 1:43pm

FYI, if you don’t mind the console always being a bit “chatty”, you could put this in “/etc/rc.local” until the issue is resolved:

/bin/echo "9" > /proc/sysrq-trigger

You may be interested in file “/var/log/kern.log”, combined with some sysrq keys. Should you have a running system fail, then the following sysrq combination could provide data in that log file:

# Call sync, though on solid state drives you do NOT want to do this often.
# However, in this case you know the system is about to be forcibly rebooted:
alt-sysrq-s
# force a kernel panic/dump with a reboot after about 30 seconds:
alt-sysrq-c

Other sysrq keys may also be useful, e.g., task list (“alt-sysrq-t”) or memory info (“alt-sysrq-m”). The serial console could be attached prior to using the sysrq keys if you want to log and observe live.

Charlinux · May 4, 2016, 7:54am

Hi linuxdev,

Thanks for the hints.
I definitely have to learn more about that.

Bibek · May 5, 2016, 9:23am

Thanks for the log.
I see so many things not correct in the log.
Can you as a first step disable wifi. I want to get rid of mmc1 dumps

mv /lib/firmware/brcm /lib/firmware/brcm_disabled

and then reboot and then provide logs

One more thing, have you applied the patch video-tegra-host-use-dynamically-allocated-wait-queue.patch.tar before running the test?

Charlinux · May 10, 2016, 8:05am

Hi,

Okay, I have disabled the wifi.
About the patch, yes it is applied.

I’ve re-run my software until freeze. Now I’m getting those logs in a continuous loop:

[ 7939.549741] BUG: soft lockup - CPU#3 stuck for 23s! [nvmap-bz:57]
[ 7939.555824] Modules linked in: cfg80211 nvhost_vi bluedroid_pm
[ 7939.561693] 
[ 7939.563183] CPU: 3 PID: 57 Comm: nvmap-bz Tainted: G        W    3.10.67-g458d45c #1
[ 7939.570912] task: ffffffc0fec3b5c0 ti: ffffffc0fecd0000 task.ti: ffffffc0fecd0000
[ 7939.578385] PC is at csd_lock_wait+0x10/0x1c
[ 7939.582646] LR is at smp_call_function_many+0x254/0x2a8
[ 7939.587860] pc : [<ffffffc000104fc8>] lr : [<ffffffc0001056e4>] pstate: 80000145
[ 7939.595239] sp : ffffffc0fecd3c90
[ 7939.598545] x29: ffffffc0fecd3c90 x28: ffffffc001120f4c 
[ 7939.603869] x27: 0000000000000000 x26: ffffffc03fedf288 
[ 7939.609194] x25: ffffffc03fedf280 x24: ffffffc03fed22d0 
[ 7939.614516] x23: 000000003ede2000 x22: ffffffc001323000 
[ 7939.619837] x21: 0000000000000008 x20: ffffffc001120f4c 
[ 7939.625160] x19: 0000000000000000 x18: 0000000000000000 
[ 7939.630484] x17: 0000000000000000 x16: 00000000000000ef 
[ 7939.635807] x15: 00000000000000e0 x14: 0000000000000000 
[ 7939.641130] x13: 0000000000000029 x12: 0000000000000000 
[ 7939.646452] x11: 0000000000000000 x10: 000000003edef000 
[ 7939.651775] x9 : ffffffc0fecd3bf0 x8 : ffffffc0fec3bae0 
[ 7939.657097] x7 : ffffffbc04f79d28 x6 : ffffffbc04f79cf0 
[ 7939.662419] x5 : ffffffc03fedf288 x4 : 0000000000000000 
[ 7939.667740] x3 : ffffffc0010f4648 x2 : 0000000000000004 
[ 7939.673062] x1 : ffffffc03febc668 x0 : 0000000000000001 
[ 7939.678386] 
[ 7939.678386] PC: 0xffffffc000104f48:
[ 7939.683340] 4f48  9426a41c 910003e0 9272c415 f9400aa0 d2800021 f9000001 d5033bbf aa1303e0
[ 7939.691605] 4f68  aa1403e2 aa1603e3 9426a1bc 2a0003f4 f9400aa0 f900001f d5033bbf 34000094
[ 7939.699871] 4f88  aa1303e0 aa1603e1 97fffec4 aa1303e0 97fffc1a aa1303e0 9426a455 2a1403e0
[ 7939.708136] 4fa8  a94153f3 a9425bf5 a8c37bfd d65f03c0 91008001 485ffc20 36000080 d503205f
[ 7939.716400] 4fc8  485ffc20 3707ffc0 d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 97fffff5
[ 7939.724666] 4fe8  79404261 32000021 79004261 d5033bbf f9400bf3 a8c27bfd d65f03c0 d65f03c0
[ 7939.732935] 5008  a9be7bfd 910003fd a90153f3 f0007f53 910a0273 d00090e0 91070000 f8627814
[ 7939.741201] 5028  8b140260 d1000c21 f100503f 540002e8 7100503f 540002a8 d0004d82 911ee042
[ 7939.749466] 
[ 7939.749466] LR: 0xffffffc000105664:
[ 7939.754419] 5664  f9000e7b 910042b8 aa1803e0 9426a374 aa0003e1 f94006a2 f90006b3 f8376a95
[ 7939.762686] 5684  f9000662 f9000053 aa1803e0 9426a3e8 110006c2 aa1a03e0 d2800101 93407c42
[ 7939.770951] 56a4  940810db 2a0003f6 b9400381 6b01001f 54fffc2b 91004320 97fe2cca 12800013
[ 7939.779217] 56c4  b9406fa0 35000100 14000014 f9400323 910702c1 f860d820 8b000060 97fffe36
[ 7939.787480] 56e4  14000005 d2800115 f00080d4 913d3294 d00090f6 11000662 aa1a03e0 aa1503e1
[ 7939.795745] 5704  93407c42 940810c2 2a0003f3 b9400282 6b02001f 54fffdcb a94153f3 a9425bf5
[ 7939.804010] 5724  a94363f7 a9446bf9 a94573fb a8c87bfd d65f03c0 a9bd7bfd 910003fd a90153f3
[ 7939.812275] 5744  f90013f5 aa0003f4 aa0103f3 2a0203f5 52800020 97ff64c4 6b1f02bf f0004d60
[ 7939.820538] 
[ 7939.820538] SP: 0xffffffc0fecd3c10:
[ 7939.825491] 3c10  01120f4c ffffffc0 00000008 00000000 01323000 ffffffc0 3ede2000 00000000
[ 7939.833759] 3c30  3fed22d0 ffffffc0 3fedf280 ffffffc0 3fedf288 ffffffc0 00000000 00000000
[ 7939.842025] 3c50  01120f4c ffffffc0 fecd3c90 ffffffc0 001056e4 ffffffc0 fecd3c90 ffffffc0
[ 7939.850293] 3c70  00104fc8 ffffffc0 80000145 00000000 fecd3c90 ffffffc0 001056c0 ffffffc0
[ 7939.858559] 3c90  fecd3d10 ffffffc0 00105778 ffffffc0 00000000 00000000 00099f00 ffffffc0
[ 7939.866822] 3cb0  00000001 00000000 feca2268 ffffffc0 01150828 ffffffc0 fecd3e18 ffffffc0
[ 7939.875089] 3cd0  00000001 00000000 fecd0000 ffffffc0 00000000 00000000 00000000 00000000
[ 7939.883356] 3cf0  fecd3d10 ffffffc0 0010575c 00000001 010f02c0 ffffffc0 00099f00 ffffffc0
[ 7939.891620] 
[ 7939.891620] X1: 0xffffffc03febc5e8:
[ 7939.896575] c5e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7939.904842] c608  00000000 00000000 00000000 00000000 00000000 00000000 00100100 00000000
[ 7939.913107] c628  00200200 00000000 00747d8c ffffffc0 00000000 00000000 00000000 00000000
[ 7939.921372] c648  3feb82c0 ffffffc0 3feb82c0 ffffffc0 00099f00 ffffffc0 00000000 00000000
[ 7939.929642] c668  00000001 00000000 00000000 00000000 00000ca8 00000000 05310be0 ffffffbc
[ 7939.937905] c688  0529d760 ffffffbc 00000005 00000004 3febc698 ffffffc0 3febc698 ffffffc0
[ 7939.946173] c6a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7939.954437] c6c8  e9b60d00 ffffffc0 fd615600 ffffffc0 00000000 00000000 00000000 0000001d
[ 7939.962704] 
[ 7939.962704] X3: 0xffffffc0010f45c8:
[ 7939.967658] 45c8  0000b7f4 04040001 ff3c5f08 0000b7ec 04040001 ff3c5f98 0000b7e4 04040001
[ 7939.975922] 45e8  ff3c6ac8 0000b7dc 04040001 ff3c6bb4 0000b7d4 04040001 ff3c876c 0000b7cc
[ 7939.984185] 4608  04040001 ff3c87fc 0000b7c4 04040001 ff3c888c 0000b7bc 04040001 ff3c8950
[ 7939.992452] 4628  0000b7b4 04040001 ff3c8a5c 0000b7ac 04040001 ff3c8b98 0000b7a4 04040001
[ 7940.000718] 4648  ff3c8c9c 0000b79c 04040001 ff3c8da4 0000b794 04040001 ff3c8ee8 0000b78c
[ 7940.008984] 4668  04040001 ff3c8fec 0000b784 04040001 ff3c9100 0000b77c 04040001 ff3c9234
[ 7940.017249] 4688  0000b774 04040001 ff3c9338 0000b76c 04040001 ff3c9454 0000b764 04040001
[ 7940.025515] 46a8  ff3c9674 0000b75c 04040001 ff3c9780 0000b754 04040001 ff3c98c8 0000b74c
[ 7940.033784] 
[ 7940.033784] X5: 0xffffffc03fedf208:
[ 7940.038737] f208  4bf3f071 000003f8 000000a1 00000020 018ef5b6 00ac123a 00d01811 00829fbc
[ 7940.047005] f228  00000000 00000000 000057bc 0000b6e7 754ede27 00000833 00000000 00000000
[ 7940.055273] f248  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.063537] f268  00000000 00000000 00000000 00000000 00000000 00000000 010f4648 ffffffc0
[ 7940.071806] f288  00000007 00000000 00000007 00000000 00000000 00000000 00000000 00000000
[ 7940.080072] f2a8  00000000 00000000 00000000 00000000 00000000 00000000 3fedf2c0 ffffffc0
[ 7940.088336] f2c8  3fedf2c0 ffffffc0 003e003e 00000000 00000000 00000000 00000000 00000000
[ 7940.096602] f2e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.104869] 
[ 7940.104869] X8: 0xffffffc0fec3ba60:
[ 7940.109823] ba60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.118088] ba80  fff60000 ffffffc0 fec3b5c0 ffffffc0 fff60000 ffffffc0 010ef8c0 ffffffc0
[ 7940.126355] baa0  00000000 00000000 010ef8c0 ffffffc0 010ef8c0 ffffffc0 01322e48 ffffffc0
[ 7940.134620] bac0  01322e68 ffffffc0 fd5e6100 ffffffc0 fecd3bf0 ffffffc0 fecd3bf0 ffffffc0
[ 7940.142888] bae0  00086b14 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.151152] bb00  7fb5d8a4 3659d30c 10c986f0 e0322c15 09026256 36b17889 84294140 25c25012
[ 7940.159418] bb20  d747762d 2712e628 de6fa324 f1280563 8b2967b1 62119934 00228078 2c605fc2
[ 7940.167680] bb40  80840936 c42d388d 1914215c 1e9c08e2 009d3d2b 12411b20 065c7201 c854b0c0
[ 7940.175948] 
[ 7940.175948] X9: 0xffffffc0fecd3b70:
[ 7940.180904] 3b70  00000001 00000000 3febc668 ffffffc0 00000004 00000000 010f4648 ffffffc0
[ 7940.189172] 3b90  00000000 00000000 3fedf288 ffffffc0 04f79cf0 ffffffbc 04f79d28 ffffffbc
[ 7940.197440] 3bb0  fec3bae0 ffffffc0 fecd3bf0 ffffffc0 3edef000 00000000 00000000 00000000
[ 7940.205705] 3bd0  00000000 00000000 00000029 00000000 00000000 00000000 000000e0 00000000
[ 7940.213970] 3bf0  000000ef 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.222240] 3c10  01120f4c ffffffc0 00000008 00000000 01323000 ffffffc0 3ede2000 00000000
[ 7940.230510] 3c30  3fed22d0 ffffffc0 3fedf280 ffffffc0 3fedf288 ffffffc0 00000000 00000000
[ 7940.238775] 3c50  01120f4c ffffffc0 fecd3c90 ffffffc0 001056e4 ffffffc0 fecd3c90 ffffffc0
[ 7940.247045] 
[ 7940.247045] X20: 0xffffffc001120ecc:
[ 7940.252085] 0ecc  00000015 00000000 00000000 01122dc0 ffffffc0 01120840 ffffffc0 000fe318
[ 7940.260350] 0eec  ffffffc0 000fe2b4 ffffffc0 00000000 00000000 00000000 00000000 00000000
[ 7940.268620] 0f0c  00000000 ffffffff 00000000 00000001 00000000 00000001 00000000 01120f28
[ 7940.276887] 0f2c  ffffffc0 01120f28 ffffffc0 00000000 00000000 00000000 00000000 00000400
[ 7940.285152] 0f4c  00000004 00000004 00000000 fc05ce48 ffffffbf fc001ee8 ffffffbf 00f361a8
[ 7940.293411] 0f6c  ffffffc0 00000080 00000000 00000000 00000000 00109fb8 ffffffc0 00000000
[ 7940.301679] 0f8c  00000000 00000000 00000000 00000000 00000000 fc000000 ffffffbf 00000000
[ 7940.309943] 0fac  000c000c 01120fb0 ffffffc0 01120fb0 ffffffc0 01123e78 ffffffc0 00000001
[ 7940.318210] 
[ 7940.318210] X22: 0xffffffc001322f80:
[ 7940.323250] 2f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.331515] 2fa0  00000000 00000000 00000000 00000000 00000006 00000004 00000001 00000000
[ 7940.339781] 2fc0  00000000 00000000 00000000 00000000 00000002 00000000 000f0dbc 00000000
[ 7940.348051] 2fe0  00000000 00000000 00001352 00000000 00001352 00000000 00000000 00000000
[ 7940.356315] 3000  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.364582] 3020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.372848] 3040  0002d802 00000000 00000000 00000000 00000000 00000000 0c2d0c2d 00000000
[ 7940.381114] 3060  ffffffec ffffffff 01128c00 ffffffc0 011215f8 ffffffc0 010f2f34 ffffffc0
[ 7940.389382] 
[ 7940.389382] X24: 0xffffffc03fed2250:
[ 7940.394422] 2250  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.402685] 2270  00000000 00000000 00000000 00000000 010f4620 ffffffc0 0000000b 00000000
[ 7940.410953] 2290  0000000b 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.419223] 22b0  00000000 00000000 00000000 00000000 3fed22c0 ffffffc0 3fed22c0 ffffffc0
[ 7940.427492] 22d0  00420042 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.435757] 22f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.444023] 2310  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.452292] 2330  00000000 00000000 00000000 00000000 00000000 00000000 3fed2340 ffffffc0
[ 7940.460555] 
[ 7940.460555] X25: 0xffffffc03fedf200:
[ 7940.465595] f200  00000000 00000000 4bf3f071 000003f8 000000a1 00000020 018ef5b6 00ac123a
[ 7940.473861] f220  00d01811 00829fbc 00000000 00000000 000057bc 0000b6e7 754ede27 00000833
[ 7940.482128] f240  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.490396] f260  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.498666] f280  010f4648 ffffffc0 00000007 00000000 00000007 00000000 00000000 00000000
[ 7940.506932] f2a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.515197] f2c0  3fedf2c0 ffffffc0 3fedf2c0 ffffffc0 003e003e 00000000 00000000 00000000
[ 7940.523463] f2e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.531731] 
[ 7940.531731] X26: 0xffffffc03fedf208:
[ 7940.536771] f208  4bf3f071 000003f8 000000a1 00000020 018ef5b6 00ac123a 00d01811 00829fbc
[ 7940.545039] f228  00000000 00000000 000057bc 0000b6e7 754ede27 00000833 00000000 00000000
[ 7940.553306] f248  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.561573] f268  00000000 00000000 00000000 00000000 00000000 00000000 010f4648 ffffffc0
[ 7940.569839] f288  00000007 00000000 00000007 00000000 00000000 00000000 00000000 00000000
[ 7940.578103] f2a8  00000000 00000000 00000000 00000000 00000000 00000000 3fedf2c0 ffffffc0
[ 7940.586374] f2c8  3fedf2c0 ffffffc0 003e003e 00000000 00000000 00000000 00000000 00000000
[ 7940.594639] f2e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7940.602908] 
[ 7940.602908] X28: 0xffffffc001120ecc:
[ 7940.607949] 0ecc  00000015 00000000 00000000 01122dc0 ffffffc0 01120840 ffffffc0 000fe318
[ 7940.616212] 0eec  ffffffc0 000fe2b4 ffffffc0 00000000 00000000 00000000 00000000 00000000
[ 7940.624478] 0f0c  00000000 ffffffff 00000000 00000001 00000000 00000001 00000000 01120f28
[ 7940.632744] 0f2c  ffffffc0 01120f28 ffffffc0 00000000 00000000 00000000 00000000 00000400
[ 7940.641012] 0f4c  00000004 00000004 00000000 fc05ce48 ffffffbf fc001ee8 ffffffbf 00f361a8
[ 7940.649279] 0f6c  ffffffc0 00000080 00000000 00000000 00000000 00109fb8 ffffffc0 00000000
[ 7940.657547] 0f8c  00000000 00000000 00000000 00000000 00000000 fc000000 ffffffbf 00000000
[ 7940.665812] 0fac  000c000c 01120fb0 ffffffc0 01120fb0 ffffffc0 01123e78 ffffffc0 00000001
[ 7940.674080] 
[ 7940.674080] X29: 0xffffffc0fecd3c10:
[ 7940.679122] 3c10  01120f4c ffffffc0 00000008 00000000 01323000 ffffffc0 3ede2000 00000000
[ 7940.687388] 3c30  3fed22d0 ffffffc0 3fedf280 ffffffc0 3fedf288 ffffffc0 00000000 00000000
[ 7940.695656] 3c50  01120f4c ffffffc0 fecd3c90 ffffffc0 001056e4 ffffffc0 fecd3c90 ffffffc0
[ 7940.703925] 3c70  00104fc8 ffffffc0 80000145 00000000 fecd3c90 ffffffc0 001056c0 ffffffc0
[ 7940.712193] 3c90  fecd3d10 ffffffc0 00105778 ffffffc0 00000000 00000000 00099f00 ffffffc0
[ 7940.720457] 3cb0  00000001 00000000 feca2268 ffffffc0 01150828 ffffffc0 fecd3e18 ffffffc0
[ 7940.728725] 3cd0  00000001 00000000 fecd0000 ffffffc0 00000000 00000000 00000000 00000000
[ 7940.736992] 3cf0  fecd3d10 ffffffc0 0010575c 00000001 010f02c0 ffffffc0 00099f00 ffffffc0

It seems to be something about the bluetooth ? Maybe I should disable that too ?

Now, the full kernel bootlogs:

Thank you for your help.

Bibek · May 14, 2016, 5:57pm

Your Pastbin post has only 47 seconds log and everything is fine there. I wish you have posted the full log.
Is there a way to reproduce the issue locally, that would expedite the process?
Meanwhile please pass on the full log till the issue occurs with loglevel set to max.

Charlinux · May 20, 2016, 3:30pm

Hi bbasu,

I’m sorry, my post wasn’t very clear: the pastbin contains only the boottime logs. When the boot was done, I launched my software and waited for the board to freeze. When I came back the next day, I found my terminal full with the previous log (soft lockup) spinning. Maybe there were something in between…

Regarding a way to reproduce the issue, I have a minimal app but it is still confidential. I sent you a private message with the details to get it.

One more thing, I think my board is one of the first batch (I don’t know how to verify that), I received it around november 2015.

Topic		Replies	Views
Tegra tk1 freez display Jetson TK1	4	586	October 18, 2021
Jetson TX2 bootup failure after reach 142F / 61C Jetson TX2	12	1384	October 18, 2021
Wireless problem on Tx1 Jetson TX1	53	17391	October 18, 2021
Jetsen TK1 stops responding with SSH connection Jetson TK1	12	2760	December 27, 2014
TX1 freeze during boot Jetson TX1	9	1769	October 18, 2021
RT patches caused system crash Jetson TX2	4	1271	October 18, 2021
tx2-4Gb PING suddenly delay is very big with eth0. Two pieces of tx2 boards with switch connection Jetson TX2 ethernet	25	1023	October 18, 2021
One frame latency/delay in TX1 V4L stack Jetson TX1	44	9534	October 18, 2021
TK1 demo board hang Jetson TK1	34	2928	October 18, 2021
System suddenly freezes？ Jetson Xavier NX kernel	7	612	July 20, 2022

TX1 board freeze

Related topics