Kernel panic: Linux tegra-allspark 5.10.192-tegra

Jetson Linux 35.5.0.

Im going to dump the entirety of the kernel panic below. If anyone can pinpoint where it might originate from, please let me know. We have LUKS encryption enabled for secure boot and OPTEE enabled.

Thanks, Jim

D3640: /u2/balson/sw/powercycle/ota > cat panic

[ 115.263544] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 115.464812] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 115.666057] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 115.867337] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 116.068614] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 116.269858] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 116.471134] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 116.672377] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 116.873619] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 117.074898] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 117.276143] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 117.477422] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 117.678699] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 117.879942] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 118.111236] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 129.963303] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 129.963549] Buffer I/O error on dev dm-1, logical block 0, lost async page write
[ 134.319400] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 134.319667] Buffer I/O error on dev dm-0, logical block 0, lost async page write
[ 134.319869] Buffer I/O error on dev dm-0, logical block 1, lost async page write
[ 134.320079] Buffer I/O error on dev dm-0, logical block 2, lost async page write
[ 134.320285] Buffer I/O error on dev dm-0, logical block 3, lost async page write
[ 134.320473] Buffer I/O error on dev dm-0, logical block 9253, lost async page write
[ 134.320665] Buffer I/O error on dev dm-0, logical block 1048577, lost async page write
[ 134.320861] Buffer I/O error on dev dm-0, logical block 1048578, lost async page write
[ 134.321061] Buffer I/O error on dev dm-0, logical block 1048579, lost async page write
[ 134.461501] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 134.461851] EXT4-fs (dm-0): previous I/O error to superblock detected
[ 134.522286] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 134.522507] Buffer I/O error on dev dm-0, logical block 1048580, lost async page write
[ 134.723825] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 134.724074] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm kworker/u16:2: Error while async write back metadata
[ 134.925157] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 135.126449] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 135.126658] buffer_io_error: 20 callbacks suppressed
[ 135.126665] Buffer I/O error on dev dm-0, logical block 1581088, lost async page write
[ 135.127019] Buffer I/O error on dev dm-0, logical block 3670016, lost async page write
[ 135.127223] Buffer I/O error on dev dm-0, logical block 3670017, lost async page write
[ 135.127423] Buffer I/O error on dev dm-0, logical block 2622551, lost async page write
[ 135.127621] Buffer I/O error on dev dm-0, logical block 3670264, lost async page write
[ 135.127815] Buffer I/O error on dev dm-0, logical block 3670265, lost async page write
[ 135.128005] Buffer I/O error on dev dm-0, logical block 3670254, lost async page write
[ 135.128196] Buffer I/O error on dev dm-0, logical block 2622513, lost async page write
[ 135.329377] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 135.329583] Buffer I/O error on dev dm-0, logical block 3670268, lost async page write
[ 135.329806] Buffer I/O error on dev dm-0, logical block 3678330, lost async page write
[ 135.531061] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 135.732339] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 135.732588] EXT4-fs (dm-0): previous I/O error to superblock detected
[ 135.933848] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 135.934075] EXT4-fs (dm-0): I/O error while writing superblock
[ 135.934080] EXT4-fs (dm-0): I/O error while writing superblock
[ 135.934133] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm kworker/u16:2: Error while async write back metadata
[ 136.058121] rcu: INFO: rcu_preempt self-detected stall on CPU
[ 136.058333] rcu: 0-…: (5249 ticks this GP) idle=292/1/0x4000000000000002 softirq=10699/11215 fqs=2306
[ 136.058688] rcu: ====For debug only: Start Printing Blocked Tasks====<print_cpu_stall>
[ 136.058959] task:kworker/u16:2 state:D stack: 0 pid: 111 ppid: 2 flags:0x00000028
[ 136.059160] Workqueue: writeback wb_workfn (flush-252:0)
[ 136.059291] Call trace:
[ 136.059357] __switch_to+0xc8/0x120
[ 136.059446] __schedule+0x318/0x980
[ 136.059540] schedule+0x78/0x110
[ 136.059627] io_schedule+0x24/0x70
[ 136.059712] bit_wait_io+0x20/0x70
[ 136.059795] __wait_on_bit+0x80/0xf0
[ 136.059901] out_of_line_wait_on_bit+0xa8/0xe0
[ 136.060009] __sync_dirty_buffer+0x114/0x1e0
[ 136.060131] ext4_commit_super+0x220/0x390
[ 136.060229] save_error_info+0x80/0xa0
[ 136.060335] __ext4_error+0x90/0x1d0
[ 136.060425] __ext4_journal_get_write_access+0x128/0x130
[ 136.060561] ext4_ext_get_access.isra.0+0x44/0x90
[ 136.060673] ext4_ext_insert_extent+0x384/0x1490
[ 136.063144] ext4_ext_map_blocks+0x53c/0x1020
[ 136.067352] ext4_map_blocks+0x194/0x5c0
[ 136.071373] ext4_writepages+0x814/0xdd0
[ 136.074878] do_writepages+0x5c/0x110
[ 136.078383] __writeback_single_inode+0x4c/0x510
[ 136.082927] writeback_sb_inodes+0x20c/0x4b0
[ 136.087215] __writeback_inodes_wb+0x58/0x110
[ 136.091934] wb_writeback+0x26c/0x410
[ 136.095521] wb_workfn+0x4ac/0x630
[ 136.098691] process_one_work+0x1c4/0x4c0
[ 136.102700] worker_thread+0x54/0x450
[ 136.106287] kthread+0x148/0x170
[ 136.109958] ret_from_fork+0x10/0x18
[ 136.113466] task:kworker/6:2 state:D stack: 0 pid: 122 ppid: 2 flags:0x00000028
[ 136.121344] Workqueue: events key_garbage_collector
[ 136.126583] Call trace:
[ 136.129121] __switch_to+0xc8/0x120
[ 136.132540] __schedule+0x318/0x980
[ 136.135734] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 136.136126] schedule+0x78/0x110
[ 136.136138] schedule_timeout+0x2dc/0x340
[ 136.142478] EXT4-fs (dm-0): I/O error while writing superblock
[ 136.145838] wait_for_completion+0x8c/0x120
[ 136.145865] __wait_rcu_gp+0x184/0x190
[ 136.145878] synchronize_rcu+0x8c/0xa0
[ 136.150268] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm kworker/u16:2: Error while async write back metadata
[ 136.155815] key_garbage_collector+0x194/0x450
[ 136.155822] process_one_work+0x1c4/0x4c0
[ 136.155848] worker_thread+0x54/0x450
[ 136.191778] kthread+0x148/0x170
[ 136.195182] ret_from_fork+0x10/0x18
[ 136.198790] task:jbd2/dm-0-8 state:D stack: 0 pid: 575 ppid: 2 flags:0x00000028
[ 136.206904] Call trace:
[ 136.209100] __switch_to+0xc8/0x120
[ 136.212869] __schedule+0x318/0x980
[ 136.216356] schedule+0x78/0x110
[ 136.219512] io_schedule+0x24/0x70
[ 136.222749] bit_wait_io+0x20/0x70
[ 136.226419] __wait_on_bit+0x80/0xf0
[ 136.229919] out_of_line_wait_on_bit+0xa8/0xe0
[ 136.233951] __wait_on_buffer+0x40/0x50
[ 136.238146] jbd2_journal_commit_transaction+0x11c0/0x1cd0
[ 136.243401] kjournald2+0xc4/0x270
[ 136.246895] kthread+0x148/0x170
[ 136.250049] ret_from_fork+0x10/0x18
[ 136.253588] task:tee-supplicant state:D stack: 0 pid: 1431 ppid: 1 flags:0x00000000
[ 136.261783] Call trace:
[ 136.264570] __switch_to+0xc8/0x120
[ 136.267987] __schedule+0x318/0x980
[ 136.271920] schedule+0x78/0x110
[ 136.274901] jbd2_log_wait_commit+0xb8/0x150
[ 136.279201] jbd2_complete_transaction+0x94/0xc0
[ 136.283738] ext4_fc_commit+0x294/0x900
[ 136.287771] ext4_sync_file+0x370/0x410
[ 136.291439] vfs_fsync_range+0x50/0x90
[ 136.295544] ext4_buffered_write_iter+0x118/0x160
[ 136.300270] ext4_file_write_iter+0x64/0x7b0
[ 136.304390] new_sync_write+0xfc/0x1a0
[ 136.308583] vfs_write+0x25c/0x390
[ 136.311994] ksys_pwrite64+0x88/0xd0
[ 136.315412] __arm64_sys_pwrite64+0x2c/0x40
[ 136.319353] el0_svc_common.constprop.0+0x80/0x1d0
[ 136.324596] do_el0_svc+0x38/0xc0
[ 136.327750] el0_svc+0x1c/0x30
[ 136.330725] el0_sync_handler+0xa8/0xb0
[ 136.334743] el0_sync+0x16c/0x180
[ 136.337999] task:nvfancontrol state:D stack: 0 pid: 1677 ppid: 1 flags:0x00000000
[ 136.346028] Call trace:
[ 136.348659] __switch_to+0xc8/0x120
[ 136.352244] __schedule+0x318/0x980
[ 136.355574] schedule+0x78/0x110
[ 136.359157] schedule_timeout+0x184/0x340
[ 136.361225] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 136.363183] wait_for_completion_timeout+0x8c/0x110
[ 136.363191] tegra_bpmp_transfer+0x198/0x3b0
[ 136.363207] tegra_bpmp_thermal_get_temp+0x7c/0xe0 [tegra_bpmp_thermal]
[ 136.369271] EXT4-fs (dm-0): I/O error while writing superblock
[ 136.373868] of_thermal_get_temp+0x40/0x60
[ 136.373875] thermal_zone_get_temp+0x78/0x160
[ 136.373884] temp_show+0x40/0x90
[ 136.402999] dev_attr_show+0x40/0x80
[ 136.406327] sysfs_kf_seq_show+0x90/0x130
[ 136.410783] kernfs_seq_show+0x48/0x60
[ 136.414289] seq_read_iter+0x1d4/0x490
[ 136.418307] kernfs_fop_read_iter+0x154/0x1d0
[ 136.422338] new_sync_read+0xfc/0x1a0
[ 136.425844] vfs_read+0x130/0x1c0
[ 136.429090] ksys_read+0x7c/0x110
[ 136.432486] __arm64_sys_read+0x28/0x40
[ 136.436522] el0_svc_common.constprop.0+0x80/0x1d0
[ 136.441407] do_el0_svc+0x38/0xc0
[ 136.444402] el0_svc+0x1c/0x30
[ 136.447537] el0_sync_handler+0xa8/0xb0
[ 136.451558] el0_sync+0x16c/0x180
[ 136.454774] task:bst state:D stack: 0 pid: 2963 ppid: 1 flags:0x00000000
[ 136.463104] Call trace:
[ 136.465646] __switch_to+0xc8/0x120
[ 136.469144] __schedule+0x318/0x980
[ 136.472476] schedule+0x78/0x110
[ 136.475970] schedule_timeout+0x2dc/0x340
[ 136.479738] wait_for_completion_killable+0x98/0x180
[ 136.484550] kernel_clone+0x2d0/0x580
[ 136.488136] __do_sys_clone+0x54/0x80
[ 136.491811] __arm64_sys_clone+0x2c/0x40
[ 136.495664] el0_svc_common.constprop.0+0x80/0x1d0
[ 136.500561] do_el0_svc+0x38/0xc0
[ 136.504145] el0_svc+0x1c/0x30
[ 136.507134] el0_sync_handler+0xa8/0xb0
[ 136.507422] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 136.510969] el0_sync+0x16c/0x180
[ 136.510981] task:systemctl state:D stack: 0 pid: 3100 ppid: 3099 flags:0x00000000
[ 136.534155] Call trace:
[ 136.536610] __switch_to+0xc8/0x120
[ 136.540106] __schedule+0x318/0x980
[ 136.543351] schedule+0x78/0x110
[ 136.546850] schedule_timeout+0x2dc/0x340
[ 136.551059] wait_for_completion+0x8c/0x120
[ 136.555248] virt_efi_get_variable+0x128/0x180
[ 136.559707] efivar_entry_size+0x6c/0xe0
[ 136.563663] efivarfs_file_read+0x88/0x140
[ 136.567669] vfs_read+0xb4/0x1c0
[ 136.570738] ksys_read+0x7c/0x110
[ 136.574319] __arm64_sys_read+0x28/0x40
[ 136.578171] el0_svc_common.constprop.0+0x80/0x1d0
[ 136.583159] do_el0_svc+0x38/0xc0
[ 136.586313] el0_svc+0x1c/0x30
[ 136.589288] el0_sync_handler+0xa8/0xb0
[ 136.593394] el0_sync+0x16c/0x180
[ 136.596807] task:snautoupdate.sh state:D stack: 0 pid: 3101 ppid: 1477 flags:0x00000000
[ 136.605028] Call trace:
[ 136.607483] __switch_to+0xc8/0x120
[ 136.610984] __schedule+0x318/0x980
[ 136.614240] schedule+0x78/0x110
[ 136.617719] io_schedule+0x24/0x70
[ 136.620875] bit_wait_io+0x20/0x70
[ 136.624286] __wait_on_bit+0x80/0xf0
[ 136.628219] out_of_line_wait_on_bit+0xa8/0xe0
[ 136.632426] do_get_write_access+0x29c/0x500
[ 136.636882] jbd2_journal_get_write_access+0x6c/0xc0
[ 136.641797] __ext4_journal_get_write_access+0x80/0x130
[ 136.646945] ext4_reserve_inode_write+0x98/0xf0
[ 136.651251] __ext4_mark_inode_dirty+0x84/0x2f0
[ 136.655788] ext4_dirty_inode+0x78/0xa0
[ 136.659897] __mark_inode_dirty+0x1d8/0x4b0
[ 136.663751] generic_update_time+0x74/0x110
[ 136.668220] inode_update_time+0x58/0x70
[ 136.672144] touch_atime+0x1c0/0x260
[ 136.675388] step_into+0x3e0/0x620
[ 136.678626] walk_component+0x78/0x1f0
[ 136.682387] link_path_walk.part.0+0x248/0x360
[ 136.686931] path_openat+0xc0/0xe30
[ 136.690608] do_filp_open+0x8c/0x110
[ 136.693937] do_open_execat+0x88/0x210
[ 136.698132] open_exec+0x40/0x60
[ 136.701289] load_elf_binary+0x23c/0x10b0
[ 136.705312] bprm_execve+0x2a0/0x670
[ 136.709258] do_execveat_common.isra.0+0x1c4/0x240
[ 136.713976] __arm64_sys_execve+0x48/0x60
[ 136.718258] el0_svc_common.constprop.0+0x80/0x1d0
[ 136.722826] do_el0_svc+0x38/0xc0
[ 136.723556] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 136.726139] el0_svc+0x1c/0x30
[ 136.726159] el0_sync_handler+0xa8/0xb0
[ 136.726169] el0_sync+0x16c/0x180
[ 136.732746] EXT4-fs (dm-0): I/O error while writing superblock
[ 136.735590] task:bst state:D stack: 0 pid: 3102 ppid: 2963 flags:0x00000000
[ 136.735598] Call trace:
[ 136.735620] __switch_to+0xc8/0x120
[ 136.735636] __schedule+0x318/0x980
[ 136.739897] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 136.742850] schedule+0x78/0x110
[ 136.742869] io_schedule+0x24/0x70
[ 136.742881] bit_wait_io+0x20/0x70
[ 136.788173] __wait_on_bit+0x80/0xf0
[ 136.791412] out_of_line_wait_on_bit+0xa8/0xe0
[ 136.796046] do_get_write_access+0x29c/0x500
[ 136.800333] jbd2_journal_get_write_access+0x6c/0xc0
[ 136.804887] __ext4_journal_get_write_access+0x80/0x130
[ 136.810321] ext4_reserve_inode_write+0x98/0xf0
[ 136.814601] __ext4_mark_inode_dirty+0x84/0x2f0
[ 136.819596] ext4_dirty_inode+0x78/0xa0
[ 136.823263] __mark_inode_dirty+0x1d8/0x4b0
[ 136.827288] generic_update_time+0x74/0x110
[ 136.831323] inode_update_time+0x58/0x70
[ 136.835331] touch_atime+0x1c0/0x260
[ 136.839008] step_into+0x3e0/0x620
[ 136.842175] walk_component+0x78/0x1f0
[ 136.845925] link_path_walk.part.0+0x248/0x360
[ 136.850469] path_openat+0xc0/0xe30
[ 136.854144] do_filp_open+0x8c/0x110
[ 136.857731] do_open_execat+0x88/0x210
[ 136.861412] bprm_execve+0x164/0x670
[ 136.865087] do_execveat_common.isra.0+0x1c4/0x240
[ 136.869911] __arm64_sys_execve+0x48/0x60
[ 136.874103] el0_svc_common.constprop.0+0x80/0x1d0
[ 136.879345] do_el0_svc+0x38/0xc0
[ 136.882339] el0_svc+0x1c/0x30
[ 136.885557] el0_sync_handler+0xa8/0xb0
[ 136.889406] el0_sync+0x16c/0x180
[ 136.892562] rcu: ====For debug only: End Printing Blocked Tasks====<print_cpu_stall>
[ 136.950000] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 136.950257] EXT4-fs (dm-0): I/O error while writing superblock
[ 136.950494] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 137.151851] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 137.152063] EXT4-fs (dm-0): I/O error while writing superblock
[ 137.320754] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 137.522082] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 137.522317] EXT4-fs (dm-0): I/O error while writing superblock
[ 137.522570] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 137.723850] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 137.724115] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 137.925389] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 138.259230] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 138.460390] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 138.661719] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 138.863044] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 139.188798] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 139.390201] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 139.591624] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 139.591845] EXT4-fs: 7 callbacks suppressed
[ 139.591852] EXT4-fs (dm-0): I/O error while writing superblock
[ 139.758641] EXT4-fs error: 6 callbacks suppressed
[ 139.758669] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 139.960119] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 139.960346] EXT4-fs (dm-0): I/O error while writing superblock
[ 139.960610] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 140.161920] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 140.162127] buffer_io_error: 38 callbacks suppressed
[ 140.162135] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 140.162449] EXT4-fs (dm-0): I/O error while writing superblock
[ 140.162649] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 140.363948] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 140.364137] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 140.364344] EXT4-fs (dm-0): I/O error while writing superblock
[ 140.519009] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 140.720335] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 140.720530] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 140.720740] EXT4-fs (dm-0): I/O error while writing superblock
[ 140.720972] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 140.922267] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 140.922452] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 140.922674] EXT4-fs (dm-0): I/O error while writing superblock
[ 140.922867] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 141.124169] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 141.124367] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 141.124584] EXT4-fs (dm-0): I/O error while writing superblock
[ 141.281222] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 141.482555] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 141.482760] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 141.482971] EXT4-fs (dm-0): I/O error while writing superblock
[ 141.483210] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 141.684500] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 141.684698] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 141.684906] EXT4-fs (dm-0): I/O error while writing superblock
[ 141.685103] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 141.886384] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 141.886575] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 141.886786] EXT4-fs (dm-0): I/O error while writing superblock
[ 142.013434] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 142.214769] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 142.214972] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 142.416309] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 142.416500] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 142.617767] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 142.974547] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 143.175871] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 143.377180] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 143.700864] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 143.902189] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 144.103485] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 144.453167] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 144.654460] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 144.654680] EXT4-fs: 10 callbacks suppressed
[ 144.654688] EXT4-fs (dm-0): I/O error while writing superblock
[ 144.856001] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 144.856227] EXT4-fs (dm-0): I/O error while writing superblock
[ 145.001814] EXT4-fs error: 11 callbacks suppressed
[ 145.001821] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 145.203259] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 145.203453] buffer_io_error: 10 callbacks suppressed
[ 145.203459] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 145.203773] EXT4-fs (dm-0): I/O error while writing superblock
[ 145.204101] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 145.405372] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 145.405572] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 145.405772] EXT4-fs (dm-0): I/O error while writing superblock
[ 145.405948] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 145.607223] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 145.607415] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 145.607616] EXT4-fs (dm-0): I/O error while writing superblock
[ 145.768220] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 145.969541] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 145.969743] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 145.969947] EXT4-fs (dm-0): I/O error while writing superblock
[ 145.970194] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 146.171488] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 146.171691] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 146.171894] EXT4-fs (dm-0): I/O error while writing superblock
[ 146.172091] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 146.373374] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 146.373563] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 146.373766] EXT4-fs (dm-0): I/O error while writing superblock
[ 146.504369] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 146.705690] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 146.705889] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 146.706108] EXT4-fs (dm-0): I/O error while writing superblock
[ 146.706371] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 146.907671] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 146.907872] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 146.908072] EXT4-fs (dm-0): I/O error while writing superblock
[ 146.908256] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 147.109521] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 147.109706] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 147.303440] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 147.504751] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 147.504949] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 147.706242] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 147.907535] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 148.248212] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 148.449552] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 148.650844] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 149.021647] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 149.222974] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 149.424300] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 149.758961] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 149.759180] EXT4-fs: 10 callbacks suppressed
[ 149.759185] EXT4-fs (dm-0): I/O error while writing superblock
[ 149.960516] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 149.960725] EXT4-fs (dm-0): I/O error while writing superblock
[ 150.161925] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 150.162164] EXT4-fs (dm-0): I/O error while writing superblock
[ 150.264790] EXT4-fs error: 11 callbacks suppressed
[ 150.264797] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 150.466274] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 150.466458] buffer_io_error: 11 callbacks suppressed
[ 150.466463] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 150.466771] EXT4-fs (dm-0): I/O error while writing superblock
[ 150.467004] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 150.668305] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 150.668488] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 150.668686] EXT4-fs (dm-0): I/O error while writing superblock
[ 150.668870] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 150.870142] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 150.870324] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 150.870528] EXT4-fs (dm-0): I/O error while writing superblock
[ 151.033135] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 151.234442] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 151.234634] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 151.234839] EXT4-fs (dm-0): I/O error while writing superblock
[ 151.235070] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 151.436338] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 151.436523] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 151.436725] EXT4-fs (dm-0): I/O error while writing superblock
[ 151.436909] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 151.638189] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 151.638371] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 151.638577] EXT4-fs (dm-0): I/O error while writing superblock
[ 151.780151] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 151.981466] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 151.981647] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 151.981844] EXT4-fs (dm-0): I/O error while writing superblock
[ 151.982064] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 152.183366] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 152.183546] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 152.183780] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 152.385087] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 152.385272] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 152.537943] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 152.739281] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 152.739485] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 152.940830] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 153.142118] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 153.505834] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 153.707209] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 153.908487] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 154.257187] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 154.458513] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 154.659792] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 154.963436] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 154.963647] EXT4-fs: 11 callbacks suppressed
[ 154.963653] EXT4-fs (dm-0): I/O error while writing superblock
[ 155.164993] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 155.165213] EXT4-fs (dm-0): I/O error while writing superblock
[ 155.366450] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 155.366662] EXT4-fs (dm-0): I/O error while writing superblock
[ 155.482168] EXT4-fs error: 11 callbacks suppressed
[ 155.482174] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 155.683598] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 155.683786] buffer_io_error: 11 callbacks suppressed
[ 155.683792] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 155.684102] EXT4-fs (dm-0): I/O error while writing superblock
[ 155.684319] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 155.885578] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 155.885747] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 155.885954] EXT4-fs (dm-0): I/O error while writing superblock
[ 155.886173] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 156.087478] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 156.087667] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 156.087867] EXT4-fs (dm-0): I/O error while writing superblock
[ 156.239435] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 156.440751] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 156.440941] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 156.441144] EXT4-fs (dm-0): I/O error while writing superblock
[ 156.441377] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 156.642649] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 156.642835] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 156.643034] EXT4-fs (dm-0): I/O error while writing superblock
[ 156.643217] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 156.844500] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 156.844682] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 156.844877] EXT4-fs (dm-0): I/O error while writing superblock
[ 156.983472] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 157.184813] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 157.185000] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 157.185209] EXT4-fs (dm-0): I/O error while writing superblock
[ 157.185441] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 157.386696] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 157.386884] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 157.387133] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 157.588417] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 157.588602] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 157.780295] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 157.981599] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 157.981789] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 158.183089] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 158.384382] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 158.771118] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 158.972405] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 159.173649] tegra-se-nvhost 15820000.se: Couldn’t get free cmdbuf
[ 159.353479] aes_buffer not available
[ 159.373843] aes_buffer not available
[ 159.394146] aes_buffer not available
[ 159.585869] aes_buffer not available
[ 159.606237] aes_buffer not available
[ 159.626550] aes_buffer not available
[ 159.834299] aes_buffer not available
[ 159.854650] aes_buffer not available
[ 159.874966] aes_buffer not available
[ 160.033659] aes_buffer not available
[ 160.033787] EXT4-fs: 17 callbacks suppressed
[ 160.033792] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.054321] aes_buffer not available
[ 160.054475] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.074768] aes_buffer not available
[ 160.074917] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.265657] aes_buffer not available
[ 160.265796] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.286187] aes_buffer not available
[ 160.286331] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.306635] aes_buffer not available
[ 160.306776] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.440459] aes_buffer not available
[ 160.440594] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.460954] aes_buffer not available
[ 160.461104] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.481437] aes_buffer not available
[ 160.481581] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.599096] EXT4-fs error: 23 callbacks suppressed
[ 160.599107] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 160.619635] aes_buffer not available
[ 160.619762] EXT4-fs (dm-0): I/O error while writing superblock
[ 160.619987] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 160.640395] aes_buffer not available
[ 160.640548] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 160.660970] aes_buffer not available
[ 160.770481] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 160.790897] aes_buffer not available
[ 160.791022] buffer_io_error: 26 callbacks suppressed
[ 160.791028] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 160.791386] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 160.811786] aes_buffer not available
[ 160.811888] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 160.812095] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 160.832495] aes_buffer not available
[ 160.832617] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 160.984193] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 161.004641] aes_buffer not available
[ 161.004749] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.005007] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 161.025385] aes_buffer not available
[ 161.025513] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.025723] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 161.046143] aes_buffer not available
[ 161.046251] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.146784] EXT4-fs error (device dm-0): ext4_check_bdev_write_error:217: comm ZMQbg/IO/0: Error while async write back metadata
[ 161.167205] aes_buffer not available
[ 161.167316] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.187700] aes_buffer not available
[ 161.187825] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.208196] aes_buffer not available
[ 161.208315] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.415116] aes_buffer not available
[ 161.415227] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[ 161.435605] aes_buffer not available
[ 161.455921] aes_buffer not available
[ 161.588568] aes_buffer not available
[ 161.608966] aes_buffer not available
[ 161.629282] aes_buffer not available
[ 161.781947] aes_buffer not available
[ 161.802345] aes_buffer not available
[ 161.822661] aes_buffer not available
[ 161.995366] aes_buffer not available
[ 162.015682] aes_buffer not available
[ 162.035998] aes_buffer not available
[ 162.183668] aes_buffer not available
[ 162.203984] aes_buffer not available
[ 162.224301] aes_buffer not available
[ 162.356945] aes_buffer not available
[ 162.377277] aes_buffer not available
[ 162.397577] aes_buffer not available
[ 162.587288] aes_buffer not available
[ 162.607603] aes_buffer not available
[ 162.627920] aes_buffer not available
[ 162.818630] aes_buffer not available
[ 162.838993] aes_buffer not available
[ 162.859309] aes_buffer not available
[ 163.024019] aes_buffer not available
[ 163.044335] aes_buffer not available
[ 163.064651] aes_buffer not available
[ 163.234307] aes_buffer not available
[ 163.254622] aes_buffer not available
[ 163.274939] aes_buffer not available
[ 163.460620] aes_buffer not available
[ 163.480936] aes_buffer not available
[ 163.501253] aes_buffer not available
[ 163.521485] aes_buffer not available
[ 163.668107] aes_buffer not available
[ 163.688439] aes_buffer not available
[ 163.708739] aes_buffer not available
[ 163.860390] aes_buffer not available
[ 163.880753] aes_buffer not available
[ 163.901070] aes_buffer not available
[ 164.018147] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u16:12:3015]
[ 164.018343] Modules linked in: lzo_rle lzo_compress zram lrdmwl_usb(OE) lrdmwl(OE) mac80211(OE) cfg80211(OE) compat(OE) loop nvgpu snd_soc_tegra186_dspk snd_soc_tegra186_asrc snd_soc_tegra210_ope snd_soc_tegra210_mvc snd_soc_tegra210_iqc snd_soc_tegra186_arad snd_soc_tegra210_afc snd_soc_tegra210_dmic snd_soc_tegra210_adx snd_soc_tegra210_admaif snd_soc_tegra210_amx snd_soc_tegra210_i2s snd_soc_tegra210_sfc snd_soc_tegra210_mixer snd_soc_tegra_pcm aes_ce_blk crypto_simd snd_hda_codec_hdmi cryptd aes_ce_cipher snd_hda_tegra ghash_ce sha2_ce ucsi_ccg snd_soc_tegra_machine_driver snd_hda_codec sha256_arm64 ftdi_sio sha1_ce snd_soc_tegra_utils typec_ucsi binfmt_misc snd_soc_simple_card_utils snd_soc_spdif_tx pwm_fan typec ina3221 usbserial xdma(O) max77620_thermal snd_soc_tegra210_ahub nct1008 tegra210_adma snd_hda_core tegra_bpmp_thermal userspace_alert spi_tegra114 spi_tegra210_qspi nvmap xt_tcpudp xt_conntrack iptable_filter xt_MASQUERADE xt_addrtype iptable_nat nf_nat nf_conntrack
[ 164.018604] nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_tables x_tables [last unloaded: mtd]
[ 164.030002] CPU: 0 PID: 3015 Comm: kworker/u16:12 Tainted: G OE 5.10.192-tegra #1
[ 164.038731] aes_buffer not available
[ 164.039179] Hardware name: NVIDIA Jetson-AGXi/Jetson, BIOS 202210.4-a5ac12d7-dirty 11/19/2024
[ 164.039198] Workqueue: efi_rts_wq efi_call_rts
[ 164.055745] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=–)
[ 164.061766] pc : 0x916d47aa57a8
[ 164.062848] aes_buffer not available
[ 164.065168] lr : 0x916d47aa3c50
[ 164.065176] sp : ffff80001029bdc0
[ 164.075412] x29: ffff80001029bdd0 x28: ffff1ab2000df400
[ 164.080665] x27: 000000000000003a x26: 0000000000000000
[ 164.086445] x25: 0000916d48c00140 x24: ffff800015af3d70
[ 164.089063] aes_buffer not available
[ 164.091513] x23: 0000000032000003 x22: 0000916d47ab0000
[ 164.100784] x21: 000000000007a120 x20: 0000916d47ab02a0
[ 164.106120] x19: 0000000044de0000 x18: 0000000000000000
[ 164.111288] x17: 0000000000000000 x16: ffff80001029c000
[ 164.117058] x15: 0000000000000000 x14: 0000000000000000
[ 164.122261] x13: 00000074006f006f x12: 0042006500720075
[ 164.128021] x11: 0000000000000001 x10: 0000000000000ab0
[ 164.133525] x9 : ffff80001029be28 x8 : 0000916d479b498c
[ 164.138695] x7 : 0000000000000000 x6 : 0000000000000000
[ 164.144112] x5 : 0000000000000000 x4 : 0000000000000000
[ 164.149448] x3 : 0000000000000001 x2 : 0000000000000000
[ 164.154787] x1 : 00000000000000a0 x0 : 00000000ffff0000
[ 164.160366] Call trace:
[ 164.162649] watchdog: ====For debug only: Start Printing Blocked Tasks====<watchdog_timer_fn>
[ 164.171480] task:kworker/u16:2 state:D stack: 0 pid: 111 ppid: 2 flags:0x00000028
[ 164.179454] Workqueue: writeback wb_workfn (flush-252:0)
[ 164.184778] Call trace:
[ 164.187153] __switch_to+0xc8/0x120
[ 164.190908] __schedule+0x318/0x980
[ 164.194148] schedule+0x78/0x110
[ 164.197818] io_schedule+0x24/0x70
[ 164.201056] bit_wait_io+0x20/0x70
[ 164.204035] __wait_on_bit+0x80/0xf0
[ 164.207711] out_of_line_wait_on_bit+0xa8/0xe0
[ 164.212271] do_get_write_access+0x29c/0x500
[ 164.216039] aes_buffer not available
[ 164.216288] jbd2_journal_get_write_access+0x6c/0xc0
[ 164.216314] __ext4_journal_get_write_access+0x80/0x130
[ 164.229934] ext4_reserve_inode_write+0x98/0xf0
[ 164.234670] __ext4_mark_inode_dirty+0x84/0x2f0
[ 164.239294] ext4_dirty_inode+0x78/0xa0
[ 164.240077] aes_buffer not available
[ 164.243232] __mark_inode_dirty+0x1d8/0x4b0
[ 164.243237] ext4_da_update_reserve_space+0x194/0x200
[ 164.243248] ext4_ext_map_blocks+0xad0/0x1020
[ 164.259862] ext4_map_blocks+0x194/0x5c0
[ 164.263880] ext4_writepages+0x814/0xdd0
[ 164.267154] aes_buffer not available
[ 164.267563] do_writepages+0x5c/0x110
[ 164.267586] __writeback_single_inode+0x4c/0x510
[ 164.279552] writeback_sb_inodes+0x20c/0x4b0
[ 164.283836] __writeback_inodes_wb+0x58/0x110
[ 164.288225] wb_writeback+0x26c/0x410
[ 164.291886] wb_workfn+0x4ac/0x630
[ 164.295558] process_one_work+0x1c4/0x4c0
[ 164.299499] worker_thread+0x54/0x450
[ 164.303256] kthread+0x148/0x170
[ 164.306237] ret_from_fork+0x10/0x18
[ 164.309833] task:kworker/6:2 state:D stack: 0 pid: 122 ppid: 2 flags:0x00000028
[ 164.318146] Workqueue: events key_garbage_collector
[ 164.323216] Call trace:
[ 164.325575] __switch_to+0xc8/0x120
[ 164.329163] __schedule+0x318/0x980
[ 164.333093] schedule+0x78/0x110
[ 164.336507] schedule_timeout+0x2dc/0x340
[ 164.340361] wait_for_completion+0x8c/0x120
[ 164.344562] __wait_rcu_gp+0x184/0x190
[ 164.348670] synchronize_rcu+0x8c/0xa0
[ 164.352432] key_garbage_collector+0x194/0x450
[ 164.356819] process_one_work+0x1c4/0x4c0
[ 164.360748] worker_thread+0x54/0x450
[ 164.364423] kthread+0x148/0x170
[ 164.367919] ret_from_fork+0x10/0x18
[ 164.371720] task:jbd2/dm-0-8 state:D stack: 0 pid: 575 ppid: 2 flags:0x00000028
[ 164.379728] Call trace:
[ 164.382356] __switch_to+0xc8/0x120
[ 164.386031] __schedule+0x318/0x980
[ 164.389618] schedule+0x78/0x110
[ 164.392600] io_schedule+0x24/0x70
[ 164.395849] bit_wait_io+0x20/0x70
[ 164.399073] __wait_on_bit+0x80/0xf0
[ 164.402573] out_of_line_wait_on_bit+0xa8/0xe0
[ 164.406875] __wait_on_buffer+0x40/0x50
[ 164.411058] jbd2_journal_commit_transaction+0x11c0/0x1cd0
[ 164.416137] kjournald2+0xc4/0x270
[ 164.419632] kthread+0x148/0x170
[ 164.423044] ret_from_fork+0x10/0x18
[ 164.426322] task:tee-supplicant state:D stack: 0 pid: 1431 ppid: 1 flags:0x00000000
[ 164.434508] Call trace:
[ 164.437307] __switch_to+0xc8/0x120
[ 164.440725] __schedule+0x318/0x980
[ 164.444399] schedule+0x78/0x110
[ 164.447639] jbd2_log_wait_commit+0xb8/0x150
[ 164.452180] jbd2_complete_transaction+0x94/0xc0
[ 164.456475] ext4_fc_commit+0x294/0x900
[ 164.460494] ext4_sync_file+0x370/0x410
[ 164.464174] vfs_fsync_range+0x50/0x90
[ 164.468294] ext4_buffered_write_iter+0x118/0x160
[ 164.472749] ext4_file_write_iter+0x64/0x7b0
[ 164.477125] new_sync_write+0xfc/0x1a0
[ 164.481061] vfs_write+0x25c/0x390
[ 164.484473] ksys_pwrite64+0x88/0xd0
[ 164.484900] aes_buffer not available
[ 164.488148] __arm64_sys_pwrite64+0x2c/0x40
[ 164.488177] el0_svc_common.constprop.0+0x80/0x1d0
[ 164.501008] do_el0_svc+0x38/0xc0
[ 164.504244] el0_svc+0x1c/0x30
[ 164.506962] el0_sync_handler+0xa8/0xb0
[ 164.510981] el0_sync+0x16c/0x180
[ 164.512129] aes_buffer not available
[ 164.514249] task:nvfancontrol state:D stack: 0 pid: 1677 ppid: 1 flags:0x00000000
[ 164.525942] Call trace:
[ 164.528313] __switch_to+0xc8/0x120
[ 164.532156] __schedule+0x318/0x980
[ 164.535398] schedule+0x78/0x110
[ 164.537901] aes_buffer not available
[ 164.538813] schedule_timeout+0x184/0x340
[ 164.538819] wait_for_completion_timeout+0x8c/0x110
[ 164.538831] tegra_bpmp_transfer+0x198/0x3b0
[ 164.555963] tegra_bpmp_thermal_get_temp+0x7c/0xe0 [tegra_bpmp_thermal]
[ 164.562697] of_thermal_get_temp+0x40/0x60
[ 164.566475] thermal_zone_get_temp+0x78/0x160
[ 164.571093] temp_show+0x40/0x90
[ 164.574422] dev_attr_show+0x40/0x80
[ 164.578008] sysfs_kf_seq_show+0x90/0x130
[ 164.582119] kernfs_seq_show+0x48/0x60
[ 164.585970] seq_read_iter+0x1d4/0x490
[ 164.589906] kernfs_fop_read_iter+0x154/0x1d0
[ 164.593936] new_sync_read+0xfc/0x1a0
[ 164.597449] vfs_read+0x130/0x1c0
[ 164.600849] ksys_read+0x7c/0x110
[ 164.604086] __arm64_sys_read+0x28/0x40
[ 164.607775] el0_svc_common.constprop.0+0x80/0x1d0
[ 164.612832] do_el0_svc+0x38/0xc0
[ 164.616243] el0_svc+0x1c/0x30
[ 164.619049] el0_sync_handler+0xa8/0xb0
[ 164.623069] el0_sync+0x16c/0x180
[ 164.626528] task:bst state:D stack: 0 pid: 2963 ppid: 1 flags:0x00000000
[ 164.634619] Call trace:
[ 164.636900] __switch_to+0xc8/0x120
[ 164.640657] __schedule+0x318/0x980
[ 164.643986] schedule+0x78/0x110
[ 164.647224] schedule_timeout+0x2dc/0x340
[ 164.651249] wait_for_completion_killable+0x98/0x180
[ 164.656318] kernel_clone+0x2d0/0x580
[ 164.659649] __do_sys_clone+0x54/0x80
[ 164.663323] __arm64_sys_clone+0x2c/0x40
[ 164.667174] el0_svc_common.constprop.0+0x80/0x1d0
[ 164.672075] do_el0_svc+0x38/0xc0
[ 164.675658] el0_svc+0x1c/0x30
[ 164.678375] el0_sync_handler+0xa8/0xb0
[ 164.682224] el0_sync+0x16c/0x180
[ 164.685812] task:systemctl state:D stack: 0 pid: 3100 ppid: 3099 flags:0x00000000
[ 164.693774] Call trace:
[ 164.696657] __switch_to+0xc8/0x120
[ 164.700074] __schedule+0x318/0x980
[ 164.700086] aes_buffer not available
[ 164.703830] schedule+0x78/0x110
[ 164.703836] schedule_timeout+0x2dc/0x340
[ 164.703854] wait_for_completion+0x8c/0x120
[ 164.718624] virt_efi_get_variable+0x128/0x180
[ 164.723098] efivar_entry_size+0x6c/0xe0
[ 164.727021] efivarfs_file_read+0x88/0x140
[ 164.727300] aes_buffer not available
[ 164.731218] vfs_read+0xb4/0x1c0
[ 164.731224] ksys_read+0x7c/0x110
[ 164.731232] __arm64_sys_read+0x28/0x40
[ 164.745055] el0_svc_common.constprop.0+0x80/0x1d0
[ 164.750207] do_el0_svc+0x38/0xc0
[ 164.753274] el0_svc+0x1c/0x30
[ 164.754825] aes_buffer not available
[ 164.756424] el0_sync_handler+0xa8/0xb0
[ 164.756433] el0_sync+0x16c/0x180
[ 164.767356] task:snautoupdate.sh state:D stack: 0 pid: 3101 ppid: 1477 flags:0x00000000
[ 164.775491] Call trace:
[ 164.777688] __switch_to+0xc8/0x120
[ 164.781188] __schedule+0x318/0x980
[ 164.784861] schedule+0x78/0x110
[ 164.788531] io_schedule+0x24/0x70
[ 164.791709] bit_wait_io+0x20/0x70
[ 164.795355] __wait_on_bit+0x80/0xf0
[ 164.798943] out_of_line_wait_on_bit+0xa8/0xe0
[ 164.803332] do_get_write_access+0x29c/0x500
[ 164.807261] jbd2_journal_get_write_access+0x6c/0xc0
[ 164.812419] __ext4_journal_get_write_access+0x80/0x130
[ 164.817250] ext4_reserve_inode_write+0x98/0xf0
[ 164.821786] __ext4_mark_inode_dirty+0x84/0x2f0
[ 164.826681] ext4_dirty_inode+0x78/0xa0
[ 164.830531] __mark_inode_dirty+0x1d8/0x4b0
[ 164.834646] generic_update_time+0x74/0x110
[ 164.838588] inode_update_time+0x58/0x70
[ 164.842780] touch_atime+0x1c0/0x260
[ 164.846284] step_into+0x3e0/0x620
[ 164.849261] walk_component+0x78/0x1f0
[ 164.853293] link_path_walk.part.0+0x248/0x360
[ 164.857568] path_openat+0xc0/0xe30
[ 164.860987] do_filp_open+0x8c/0x110
[ 164.864832] do_open_execat+0x88/0x210
[ 164.868768] open_exec+0x40/0x60
[ 164.871928] load_elf_binary+0x23c/0x10b0
[ 164.875950] bprm_execve+0x2a0/0x670
[ 164.879882] do_execveat_common.isra.0+0x1c4/0x240
[ 164.884613] __arm64_sys_execve+0x48/0x60
[ 164.888652] el0_svc_common.constprop.0+0x80/0x1d0
[ 164.893706] do_el0_svc+0x38/0xc0
[ 164.896774] el0_svc+0x1c/0x30
[ 164.899938] el0_sync_handler+0xa8/0xb0
[ 164.904118] el0_sync+0x16c/0x180
[ 164.907532] task:bst state:D stack: 0 pid: 3102 ppid: 2963 flags:0x00000000
[ 164.915842] Call trace:
[ 164.918212] __switch_to+0xc8/0x120
[ 164.921968] __schedule+0x318/0x980
[ 164.925644] schedule+0x78/0x110
[ 164.928881] io_schedule+0x24/0x70
[ 164.931962] bit_wait_io+0x20/0x70
[ 164.935186] __wait_on_bit+0x80/0xf0
[ 164.938687] out_of_line_wait_on_bit+0xa8/0xe0
[ 164.943243] do_get_write_access+0x29c/0x500
[ 164.947608] jbd2_journal_get_write_access+0x6c/0xc0
[ 164.952418] __ext4_journal_get_write_access+0x80/0x130
[ 164.957505] ext4_reserve_inode_write+0x98/0xf0
[ 164.959741] aes_buffer not available
[ 164.962043] __ext4_mark_inode_dirty+0x84/0x2f0
[ 164.962053] ext4_dirty_inode+0x78/0xa0
[ 164.974032] __mark_inode_dirty+0x1d8/0x4b0
[ 164.978232] generic_update_time+0x74/0x110
[ 164.982431] inode_update_time+0x58/0x70
[ 164.985596] aes_buffer not available
[ 164.986193] touch_atime+0x1c0/0x260
[ 164.993196] step_into+0x3e0/0x620
[ 164.996174] walk_component+0x78/0x1f0
[ 165.000368] link_path_walk.part.0+0x248/0x360
[ 165.004487] path_openat+0xc0/0xe30
[ 165.008434] do_filp_open+0x8c/0x110
[ 165.009678] aes_buffer not available
[ 165.012093] do_open_execat+0x88/0x210
[ 165.012098] bprm_execve+0x164/0x670
[ 165.012117] do_execveat_common.isra.0+0x1c4/0x240
[ 165.012128] __arm64_sys_execve+0x48/0x60
[ 165.032134] el0_svc_common.constprop.0+0x80/0x1d0
[ 165.036943] do_el0_svc+0x38/0xc0
[ 165.040012] el0_svc+0x1c/0x30
[ 165.043419] el0_sync_handler+0xa8/0xb0
[ 165.047098] el0_sync+0x16c/0x180
[ 165.050768] task:kworker/u17:3 state:D stack: 0 pid: 3103 ppid: 2 flags:0x00000028
[ 165.058891] Workqueue: phy1 ieee80211_scan_work [mac80211]
[ 165.064503] Call trace:
[ 165.066701] __switch_to+0xc8/0x120
[ 165.070632] __schedule+0x318/0x980
[ 165.074130] schedule+0x78/0x110
[ 165.077286] schedule_timeout+0x2dc/0x340
[ 165.081311] wait_for_completion+0x8c/0x120
[ 165.085597] __wait_rcu_gp+0x184/0x190
[ 165.089445] synchronize_rcu+0x8c/0xa0
[ 165.093450] __ieee80211_scan_completed+0x98/0x4e0 [mac80211]
[ 165.098716] ieee80211_scan_work+0x15c/0x810 [mac80211]
[ 165.103628] process_one_work+0x1c4/0x4c0
[ 165.107993] worker_thread+0x54/0x450
[ 165.111323] kthread+0x148/0x170
[ 165.114906] ret_from_fork+0x10/0x18
[ 165.118236] watchdog: ====For debug only: End Printing Blocked Tasks====<watchdog_timer_fn>
[ 165.126892] Kernel panic - not syncing: softlockup: hung tasks
[ 165.132502] CPU: 0 PID: 3015 Comm: kworker/u16:12 Tainted: G OEL 5.10.192-tegra #1
[ 165.141328] Hardware name: NVIDIA Jetson-AGXi/Jetson, BIOS 202210.4-a5ac12d7-dirty 11/19/2024
[ 165.147959] aes_buffer not available
[ 165.149410] Workqueue: efi_rts_wq efi_call_rts
[ 165.153368] EXT4-fs: 65 callbacks suppressed
[ 165.153374] EXT4-fs (dm-0): I/O error while writing superblock
[ 165.157604] Call trace:
[ 165.157617] dump_backtrace+0x0/0x1e0
[ 165.157644] SMP: stopping secondary CPUs
[ 165.177762] Kernel Offset: 0x2aaccfee0000 from 0xffff800010000000
[ 165.183770] PHYS_OFFSET: 0xffffe54f00000000
[ 165.188229] CPU features: 0x48240002,03802a30
[ 165.192692] Memory Limit: none
[ 165.195847] —[ end Kernel panic - not syncing: softlockup: hung tasks ]—

The first messages output come from the following driver source file:

    kernel/nvidia/drivers/crypto/tegra-se-nvhost.c

This is the only place these message are coming from as far as I can see. I’ve searched every file under sources/kernel and the file tegra-se-nvhost.c is the only file that comes up. Something to do with encryption for sure. But other than that at the moment, I’m looking for clues as to why …

Any clues much apprecieated.

Jim

hello JB3,

I assume it’s due to an incorrect EKS image, or, it’s failure to unlock the encrypted device. such failure should reported in the bootloader logs, and also very beginning of kernel message.
hence, please try setup serial console to gather the complete UART logs for reference.