My environment is :
32.7.3 BSP
Xavier NX 8GB module and customized carried board.
I am running a reboot test and will execute the ‘reboot’ command to restart the system every 60 seconds.
But the system occasionally freezes.
Here are the logs of the two instances where I encountered the problem
reboot测试卡死log1.txt (152.5 KB)
reboot测试卡死log2.txt (126.2 KB)
可以請問一下為何你拿著一個Xavier NX在rel-32的log然後在說rel-35.3.1 Orin NX的狀況…??
不好意思 我寫混了 我現在改回來…
請問能拿同一顆module在NV devkit上做一樣的測試複製到這個問題嗎?
我還沒有做過這個測試，我現在重新驗證一下看看
也請確認一下你們的kernel裡面這兩個patch還在不在
commit 3ada968bb16f811c601d185b22ee4107d04f7761 block: do not allow updates through sysfs until registration completes
commit b32b6b3a7dfe012a766aced4e74ee1b4f988de98 blk-mq: Register <dev>/queue/mq after having registered <dev>/queue
如果有的話請revert掉
請問這兩個commit在哪裏查看？
我們沒有拿到貴司的git倉庫路徑，都是直接下載的bsp壓縮包
你如果用我們的source_sync.sh tool抓 kernel source code下來的話是會有git log可以看的.
嗨，WayneWWW，
经过检查，这两个提交 BSP内已经回退了。
Hi，WayneWWW：
我们这边开发板损坏了，所以暂时没办法验证开发板。
但是经过经过对比 我发现重启卡死的log前有相关异常报错：
Sep 6 03:18:40 tegra-ubuntu snapd[5129]: stateengine.go:149: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/sections: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Sep 6 03:18:42 tegra-ubuntu snapd[5129]: daemon.go:521: gracefully waiting for running hooks
Sep 6 03:18:42 tegra-ubuntu snapd[5129]: daemon.go:523: done waiting for running hooks
Sep 6 03:18:42 tegra-ubuntu snapd[5129]: overlord.go:504: Released state lock file
Sep 6 03:18:42 tegra-ubuntu snapd[5129]: daemon stop requested to wait for socket activation
Sep 6 03:19:08 tegra-ubuntu systemd-udevd[2592]: seq 4011 '/devices/virtual/block/loop0' is taking a long time
Sep 6 03:21:08 tegra-ubuntu systemd-udevd[2592]: seq 4011 '/devices/virtual/block/loop0' killed
Sep 6 03:21:59 tegra-ubuntu kernel: [ 243.040585] INFO: task systemd:1 blocked for more than 120 seconds.
Sep 6 03:21:59 tegra-ubuntu kernel: [ 243.063564] Not tainted 4.9.299 #113
Sep 6 03:21:59 tegra-ubuntu kernel: [ 243.076408] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
...
对比正常重启的log，没有相关的信息，能帮忙分析下是否可能是snapd导致的问题吗？
正常重启测试syslog.txt (264.7 KB)
重启测试卡死syslog.txt (192.5 KB)
我们又复现了一次卡死的情况，卡死的log与上方提供的报错log基本一致
另外，在卡死时调试串口输入是没有响应的，HDMI显示的画面也是报错的log，
但此时拔插usb设备能显示usb的相关log信息，说明内核没有死掉，是否可能是某些服务异常导致上层卡死？
Sep 6 03:18:17 tegra-ubuntu kernel: [ 20.325417] Bluetooth: hci0: HCI_OP_READ_LOCAL_VERSION failed (-110)
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.246610] vdd-sdmmc1-sw: disabling
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.247143] vdd-1v8-sd: disabling
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.247475] vdd-1v8-cvb: disabling
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.247751] vdd-epb-1v0: disabling
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.248279] avdd-cam-2v8: disabling
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.249155] vdd-fan: disabling
Sep 6 03:18:35 tegra-ubuntu kernel: [ 38.249833] vdd_sys_en: disabling
Sep 6 03:18:39 tegra-ubuntu snapd[4778]: stateengine.go:149: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/sections: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Sep 6 03:18:42 tegra-ubuntu snapd[4778]: daemon.go:521: gracefully waiting for running hooks
Sep 6 03:18:42 tegra-ubuntu snapd[4778]: daemon.go:523: done waiting for running hooks
Sep 6 03:18:42 tegra-ubuntu snapd[4778]: overlord.go:504: Released state lock file
Sep 6 03:18:42 tegra-ubuntu snapd[4778]: daemon stop requested to wait for socket activation
Sep 6 03:19:07 tegra-ubuntu systemd-udevd[2559]: seq 3957 '/devices/virtual/block/loop0' is taking a long time
Sep 6 03:19:07 tegra-ubuntu systemd-udevd[2559]: seq 3960 '/devices/virtual/block/loop3' is taking a long time
Sep 6 03:19:08 tegra-ubuntu systemd-udevd[2559]: seq 4667 '/devices/virtual/block/loop1' is taking a long time
Sep 6 03:21:07 tegra-ubuntu systemd-udevd[2559]: seq 3957 '/devices/virtual/block/loop0' killed
Sep 6 03:21:07 tegra-ubuntu systemd-udevd[2559]: seq 3960 '/devices/virtual/block/loop3' killed
Sep 6 03:21:08 tegra-ubuntu systemd-udevd[2559]: seq 4667 '/devices/virtual/block/loop1' killed
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.045359] INFO: task systemd:1 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.046066] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.046424] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.047276] systemd D 0 1 0 0x00000000
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.047836] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.048197] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.051451] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.051902] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.054194] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.054497] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.054787] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.055147] [<ffffff80080b400c>] get_online_cpus.part.8+0x2c/0x50
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.055424] [<ffffff80080b4058>] get_online_cpus+0x28/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.055686] [<ffffff80080e7470>] tg_set_cfs_bandwidth+0x70/0x288
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.057437] [<ffffff80080e7748>] cpu_cfs_period_write_u64+0x50/0x68
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.057881] [<ffffff8008163bf4>] cgroup_file_write+0xbc/0x100
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.058600] [<ffffff80082f3b9c>] kernfs_fop_write+0xfc/0x1e0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.059941] [<ffffff800825b598>] __vfs_write+0x48/0x118
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.060187] [<ffffff800825c58c>] vfs_write+0xac/0x1b0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.060743] [<ffffff800825dc14>] SyS_write+0x5c/0xc8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.064540] [<ffffff8008083900>] el0_svc_naked+0x34/0x38
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.069870] INFO: task kworker/u12:0:6 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.076175] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.080101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.088356] kworker/u12:0 D 0 6 2 0x00000000
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.094056] Workqueue: memcg_kmem_cache_create memcg_kmem_cache_create_func
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.100856] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.103484] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.109411] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.113940] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.119077] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.125280] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.132104] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.136748] [<ffffff80080b400c>] get_online_cpus.part.8+0x2c/0x50
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.143046] [<ffffff80080b4058>] get_online_cpus+0x28/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.148185] [<ffffff80081f40f0>] memcg_create_kmem_cache+0x28/0x170
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.154667] [<ffffff8008247acc>] memcg_kmem_cache_create_func+0x2c/0x100
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.161228] [<ffffff80080d4154>] process_one_work+0x1e4/0x4b0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.166752] [<ffffff80080d4470>] worker_thread+0x50/0x4c8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.172012] [<ffffff80080db154>] kthread+0xec/0xf0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.177523] [<ffffff80080838a0>] ret_from_fork+0x10/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.182856] INFO: task kworker/4:0:36 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.189249] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.193215] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.201734] kworker/4:0 D 0 36 2 0x00000000
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.207055] Workqueue: events key_garbage_collector
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.211518] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.214220] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.219418] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.224977] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.229772] [<ffffff8008f8b294>] schedule_timeout+0x1ec/0x420
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.235153] [<ffffff8008f88cc0>] wait_for_common+0xa0/0x140
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.241073] [<ffffff8008f88d88>] wait_for_completion+0x28/0x38
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.246907] [<ffffff800812c024>] __wait_rcu_gp+0x104/0x138
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.251975] [<ffffff800812f5f8>] synchronize_rcu+0x80/0x90
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.258089] [<ffffff80083ec958>] key_garbage_collector+0x198/0x390
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.264072] [<ffffff80080d4154>] process_one_work+0x1e4/0x4b0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.269484] [<ffffff80080d4470>] worker_thread+0x50/0x4c8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.274997] [<ffffff80080db154>] kthread+0xec/0xf0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.279804] [<ffffff80080838a0>] ret_from_fork+0x10/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.285402] INFO: task kworker/1:1:820 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.292175] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.296000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.303945] kworker/1:1 D 0 820 2 0x00000000
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.309538] Workqueue: events vmstat_shepherd
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.313568] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.316366] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.321617] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.326870] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.331513] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.338342] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.344544] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.349620] [<ffffff80080b400c>] get_online_cpus.part.8+0x2c/0x50
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.355982] [<ffffff80080b4058>] get_online_cpus+0x28/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.361496] [<ffffff80081ebdac>] vmstat_shepherd+0x44/0xf8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.366762] [<ffffff80080d4154>] process_one_work+0x1e4/0x4b0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.372474] [<ffffff80080d4470>] worker_thread+0x50/0x4c8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.377454] [<ffffff80080db154>] kthread+0xec/0xf0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.382427] [<ffffff80080838a0>] ret_from_fork+0x10/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.387860] INFO: task kworker/4:1:1178 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.394952] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.398794] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.407128] kworker/4:1 D 0 1178 2 0x00000000
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.412414] Workqueue: events cpuset_hotplug_workfn
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.417142] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.419765] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.424892] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.430119] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.434861] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.441897] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.447839] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.452761] [<ffffff80080b400c>] get_online_cpus.part.8+0x2c/0x50
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.458833] [<ffffff80080b4058>] get_online_cpus+0x28/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.464564] [<ffffff80081705a8>] rebuild_sched_domains+0x20/0x70
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.470367] [<ffffff800817096c>] cpuset_hotplug_workfn+0x374/0x5d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.476454] [<ffffff80080d4154>] process_one_work+0x1e4/0x4b0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.482281] [<ffffff80080d4470>] worker_thread+0x50/0x4c8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.487434] [<ffffff80080db154>] kthread+0xec/0xf0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.492333] [<ffffff80080838a0>] ret_from_fork+0x10/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.498024] INFO: task systemd-udevd:3600 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.505008] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.508921] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.516317] systemd-udevd D 0 3600 2559 0x00000801
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.522151] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.524538] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.529871] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.535122] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.540114] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.546584] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.552886] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.557791] [<ffffff80087b94c0>] lo_release+0x28/0xc8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.562864] [<ffffff80082a3270>] __blkdev_put+0x218/0x288
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.568644] [<ffffff80082a379c>] blkdev_put+0x54/0x148
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.573674] [<ffffff80082a38bc>] blkdev_close+0x2c/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.578698] [<ffffff800825efd8>] __fput+0x90/0x1d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.583683] [<ffffff800825f190>] ____fput+0x20/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.588329] [<ffffff80080d8e6c>] task_work_run+0xbc/0xd8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.594026] [<ffffff800808b32c>] do_notify_resume+0xa4/0xb0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.599544] [<ffffff800808379c>] work_pending+0x8/0x10
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.604658] INFO: task systemd-udevd:3850 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.611872] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.615880] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.623337] systemd-udevd D 0 3850 2559 0x00000801
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.628996] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.631602] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.636549] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.641959] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.646946] [<ffffff80084390bc>] blk_mq_freeze_queue_wait+0x5c/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.653091] [<ffffff800843bd50>] blk_mq_freeze_queue+0x28/0x38
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.659025] [<ffffff80087b9554>] lo_release+0xbc/0xc8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.663842] [<ffffff80082a3270>] __blkdev_put+0x218/0x288
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.669176] [<ffffff80082a379c>] blkdev_put+0x54/0x148
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.674333] [<ffffff80082a38bc>] blkdev_close+0x2c/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.679846] [<ffffff800825efd8>] __fput+0x90/0x1d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.684910] [<ffffff800825f190>] ____fput+0x20/0x30
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.689743] [<ffffff80080d8e6c>] task_work_run+0xbc/0xd8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.695064] [<ffffff800808b32c>] do_notify_resume+0xa4/0xb0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.700610] [<ffffff800808379c>] work_pending+0x8/0x10
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.705486] INFO: task systemd-udevd:4639 blocked for more than 120 seconds.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.712749] Not tainted 4.9.299 #113
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.716938] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.724568] systemd-udevd D 0 4639 2559 0x00000801
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.730241] Call trace:
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.732607] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.738030] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.742939] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.748182] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.754446] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.760728] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.765724] [<ffffff80087b63d8>] lo_open+0x28/0x68
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.770462] [<ffffff80082a3384>] __blkdev_get+0xa4/0x468
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.775835] [<ffffff80082a3b60>] blkdev_get+0x140/0x360
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.781089] [<ffffff80082a3e7c>] blkdev_open+0x84/0x98
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.786247] [<ffffff80082590e0>] do_dentry_open+0x1d8/0x338
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.792024] [<ffffff800825a690>] vfs_open+0x58/0x88
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.796897] [<ffffff800826dd10>] do_last+0x530/0xf98
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.802028] [<ffffff800826e808>] path_openat+0x90/0x378
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.806775] [<ffffff800826fd78>] do_filp_open+0x70/0xe8
Sep 6 03:22:00 tegra-ubuntu kernel: [ 243.812004] [<ffffff800825ab54>] do_sys_open+0x174/0x258
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.817665] [<ffffff800825acbc>] SyS_openat+0x3c/0x50
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.822471] [<ffffff800808395c>] __sys_trace_return+0x0/0x4
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.828097] INFO: task nvpmodel:5965 blocked for more than 120 seconds.
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.835050] Not tainted 4.9.299 #113
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.839059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.846638] nvpmodel D 0 5965 1 0x00000008
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.852372] Call trace:
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.854992] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.860106] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.865403] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.870502] [<ffffff80084390bc>] blk_mq_freeze_queue_wait+0x5c/0xa8
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.876388] [<ffffff800843c400>] blk_mq_queue_reinit_work+0x78/0x128
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.882856] [<ffffff800843c4d4>] blk_mq_queue_reinit_dead+0x24/0x30
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.888802] [<ffffff80080b2648>] cpuhp_invoke_callback+0x98/0x5a0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.894934] [<ffffff80080b39d0>] cpuhp_down_callbacks+0x60/0xb0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.900796] [<ffffff8008f839a4>] _cpu_down+0xcc/0x180
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.906206] [<ffffff80080b48c0>] do_cpu_down+0x50/0x1c0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.911206] [<ffffff80080b4a54>] cpu_down+0x24/0x30
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.916101] [<ffffff8008787950>] cpu_subsys_offline+0x20/0x30
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.921930] [<ffffff8008780fd4>] device_offline+0x84/0xd8
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.927142] [<ffffff800878112c>] online_store+0x4c/0xa0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.932269] [<ffffff800877dd4c>] dev_attr_store+0x44/0x60
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.937930] [<ffffff80082f4910>] sysfs_kf_write+0x58/0x80
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.943133] [<ffffff80082f3b9c>] kernfs_fop_write+0xfc/0x1e0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.948736] [<ffffff800825b598>] __vfs_write+0x48/0x118
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.953983] [<ffffff800825c58c>] vfs_write+0xac/0x1b0
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.959237] [<ffffff800825dc14>] SyS_write+0x5c/0xc8
Sep 6 03:22:01 tegra-ubuntu kernel: [ 243.964049] [<ffffff8008083900>] el0_svc_naked+0x34/0x38
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.925755] INFO: task systemd:1 blocked for more than 120 seconds.
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.926713] Not tainted 4.9.299 #113
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.927485] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.928434] systemd D 0 1 0 0x00000000
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.930364] Call trace:
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.930927] [<ffffff800808604c>] __switch_to+0x9c/0xc0
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.931652] [<ffffff8008f87a98>] __schedule+0x238/0x7d0
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.932450] [<ffffff8008f88070>] schedule+0x40/0xa8
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.933769] [<ffffff8008f88520>] schedule_preempt_disabled+0x28/0x40
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.934400] [<ffffff8008f89c9c>] __mutex_lock_slowpath+0xcc/0x188
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.935029] [<ffffff8008f89da4>] mutex_lock+0x4c/0x60
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.935373] [<ffffff80080b400c>] get_online_cpus.part.8+0x2c/0x50
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.935948] [<ffffff80080b4058>] get_online_cpus+0x28/0x30
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.936333] [<ffffff80080e7470>] tg_set_cfs_bandwidth+0x70/0x288
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.937330] [<ffffff80080e7748>] cpu_cfs_period_write_u64+0x50/0x68
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.937796] [<ffffff8008163bf4>] cgroup_file_write+0xbc/0x100
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.938150] [<ffffff80082f3b9c>] kernfs_fop_write+0xfc/0x1e0
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.938475] [<ffffff800825b598>] __vfs_write+0x48/0x118
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.938685] [<ffffff800825c58c>] vfs_write+0xac/0x1b0
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.939833] [<ffffff800825dc14>] SyS_write+0x5c/0xc8
Sep 6 03:24:03 tegra-ubuntu kernel: [ 365.945273] [<ffffff8008083900>] el0_svc_naked+0x34/0x38
Sep 6 04:17:01 tegra-ubuntu CRON[6221]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 6 05:17:01 tegra-ubuntu CRON[6279]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 6 05:35:11 tegra-ubuntu kernel: [ 8234.687670] edid invalid
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.835190] tegradc 15200000.nvdisplay: blank - powerdown
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.877405] tegradc 15200000.nvdisplay: unblank
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.878593] tegra_nvdisp_handle_pd_enable: Unpowergated Head0 pd
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.878835] tegra_nvdisp_handle_pd_enable: Unpowergated Head1 pd
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.884829] Parent Clock set for DC plld2
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.892803] tegradc 15200000.nvdisplay: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_111m_223m
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.894538] tegradc 15200000.nvdisplay: hdmi: get YCC quant from EDID.
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.930881] extcon-disp-state external-connection:disp-state: cable 47 state 1
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.930884] Extcon AUX1(HDMI) enable
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.944646] tegradc 15200000.nvdisplay: sync windows ret = 246
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.973388] extcon-disp-state external-connection:disp-state: cable 51 state 1
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.973571] Extcon HDMI: HPD enabled
Sep 6 05:35:12 tegra-ubuntu kernel: [ 8234.973704] tegradc 15200000.nvdisplay: hdmi: plugged
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.228708] usb 1-2: new low-speed USB device number 5 using tegra-xusb
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.259150] usb 1-2: New USB device found, idVendor=c0f4, idProduct=01e0
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.260449] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.262270] usb 1-2: Product: usb keyboard
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.263043] usb 1-2: Manufacturer: USB
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.272975] input: USB usb keyboard as /devices/3610000.xhci/usb1/1-2/1-2:1.0/0003:C0F4:01E0.0001/input/input5
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.336341] hid-generic 0003:C0F4:01E0.0001: input,hidraw0: USB HID v1.10 Keyboard [USB usb keyboard] on usb-3610000.xhci-2/input0
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.341640] input: USB usb keyboard as /devices/3610000.xhci/usb1/1-2/1-2:1.1/0003:C0F4:01E0.0002/input/input6
Sep 6 05:35:57 tegra-ubuntu kernel: [ 8280.402439] hid-generic 0003:C0F4:01E0.0002: input,hidraw1: USB HID v1.10 Device [USB usb keyboard] on usb-3610000.xhci-2/input1
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.400687] usb 3-1.2: new low-speed USB device number 3 using xhci_hcd
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.441595] usb 3-1.2: New USB device found, idVendor=046d, idProduct=c077
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.443196] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.444307] usb 3-1.2: Product: USB Optical Mouse
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.449381] usb 3-1.2: Manufacturer: Logitech
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.465180] input: Logitech USB Optical Mouse as /devices/141a0000.pcie/pci0005:00/0005:00:00.0/0005:01:00.0/0005:02:02.0/0005:04:00.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:046D:C077.0003/input/input7
Sep 6 05:36:10 tegra-ubuntu kernel: [ 8293.468356] hid-generic 0003:046D:C077.0003: input,hidraw2: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0005:04:00.0-1.2/input0
reboot测试卡死 拔插usb有log.txt (186.1 KB)
請問你有沒有具體在NVIDIA devkit上可以複製這個問題的步驟？ 比方說一直重開機？
有需要跑其他額外的tool嗎?
HDMI显示的画面也是报错的log
請問一下這個你是看到什麼畫面？
这是我们看到的画面：
另外这是我们的重启测试方法：
1.增加/etc/rc.local文件
vim /etc/rc.local
---
#!/bin/sh -e
/etc/count.sh
2.加入/etc/count.sh脚本
每次进系统后 30s 执行reboot命令
vim /etc/count.sh
---
#!/bin/bash
echo `date +'%Y/%m/%d %A %T'`>> /home/linux/Desktop/log.txt
sleep 30 && reboot &
圖片好像沒傳上來？
已经补充在上楼了 请看下
請問大概開幾個loop會碰上這個問題？
目前测试结果，大概400~600次重启会复现一次卡死，复现时间较久
請你在kernel加一下這個看一下有沒有印些額外的
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index fb00cf30abd1..1b014294a08f 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -118,6 +118,8 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
" disables this message.\n");
sched_show_task(t);
debug_show_all_locks();
+ pr_crit("====waynewww: Blocked tasks====<%s>\n", __func__);
+ show_state_filter(TASK_UNINTERRUPTIBLE);
}
touch_nmi_watchdog();
我們這邊也會複製看看問題, 但這一個應該會蠻花時間的.