Kernel thread blocking due to nvvrs-pseq-rtc IRQs

We have encountered a problem related to the RTC driver (nvvrs-pseq-rtc). The issue doesn’t happen consistently, and we do not have clear steps on how to reproduce, but we have encountered it several times.

The issue causes a kernel thread to be blocked on i2c-call, it seems that it is trying to acquire a mutex but never actually succeeds. We added some trace and debugged the issue with following observations:

  • When everything works normally, i2c-adapter gets only calls from our thermal zone temperature readings (to the slave address 0x4c).
  • When the thread doing the temperature reading gets stuck, dmesg alerts from blocking. It also shows that i2c-adapter is flooded with calls to the power sequencer (to the slave address 0x3c)
  • Dmesg shows that the power sequencer driver calls nvvrs_pseq_irq_clear, but it doesn’t seem to get to part where it clears the interrupts (no i2c write seen on the call trace). At the same time dmesg shows that nvvrs_rtc_irq_handler gets called. It disables the alarm and succeeds in i2c writes, but the irq handler gets called repeatedly regardless.
  • When the stuck was on, we also tried to read the thermal zone temperature from terminal (cat /sys/class/thermal/thermal_zone10/temp). This caused terminal to hang.

Kernel trace from blocking alert:
11:00:34 kernel: INFO: task xxxxxxxxxx:1069 blocked for more than 120 seconds.
11:00:34 kernel: Not tainted 5.10.192+ #225
11:00:34 kernel: “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
11:00:34 kernel: task: xxxxxxxxxx state:D stack: 0 pid: 1069 ppid: 1 flags:0x00000000
11:00:34 kernel: Call trace:
11:00:34 kernel: __switch_to+0xc8/0x120
11:00:34 kernel: __schedule+0x318/0x980
11:00:34 kernel: schedule+0x78/0x108
11:00:34 kernel: __rt_mutex_slowlock+0x64/0x120
11:00:34 kernel: rt_mutex_slowlock+0xbc/0x1a0
11:00:34 kernel: rt_mutex_lock+0x84/0x90
11:00:34 kernel: i2c_adapter_lock_bus+0x24/0x30
11:00:34 kernel: i2c_smbus_xfer+0x64/0x120
11:00:34 kernel: i2c_smbus_read_byte_data+0x54/0x88
11:00:34 kernel: nct1008_read_reg+0x48/0xa0 [nct1008]
11:00:34 kernel: nct1008_get_temp_common+0x3c/0x120 [nct1008]
11:00:34 kernel: nct1008_loc_get_temp_as_sensor+0x30/0x40 [nct1008]
11:00:34 kernel: of_thermal_get_temp+0x40/0x58
11:00:34 kernel: thermal_zone_get_temp+0x78/0x158
11:00:34 kernel: temp_show+0x40/0x88
11:00:34 kernel: dev_attr_show+0x40/0x80
11:00:34 kernel: sysfs_kf_seq_show+0x90/0x130
11:00:34 kernel: kernfs_seq_show+0x48/0x58
11:00:34 kernel: seq_read_iter+0x1d4/0x490
11:00:34 kernel: kernfs_fop_read_iter+0x154/0x1c8
11:00:34 kernel: new_sync_read+0xfc/0x198
11:00:34 kernel: vfs_read+0x130/0x1b8
11:00:34 kernel: ksys_read+0x7c/0x108
11:00:34 kernel: __arm64_sys_read+0x28/0x38
11:00:34 kernel: el0_svc_common.constprop.0+0x80/0x1c8
11:00:34 kernel: do_el0_svc+0x38/0xb8
11:00:34 kernel: el0_svc+0x1c/0x28
11:00:34 kernel: el0_sync_handler+0xa8/0xb0
11:00:34 kernel: el0_sync+0x16c/0x180

Kernel trace snippet from flooding irqs. This sequence seems to be repeating:
[ 6889.487280] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.487282] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.487284] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.487285] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.487288] Call trace:
[ 6889.487290] dump_backtrace+0x0/0x1e0
[ 6889.487294] show_stack+0x30/0x40
[ 6889.487299] dump_stack+0xf0/0x130
[ 6889.487303] i2c_smbus_xfer+0x1ac/0x270
[ 6889.487305] i2c_smbus_read_byte_data+0x54/0x88
[ 6889.487309] nvvrs_pseq_irq_clear+0x6c/0x140
[ 6889.487311] regmap_irq_thread+0x37c/0x5c8
[ 6889.487314] irq_thread_fn+0x34/0xb0
[ 6889.487320] irq_thread+0x16c/0x2e0
[ 6889.487324] kthread+0x148/0x168
[ 6889.487325] ret_from_fork+0x10/0x18
[ 6889.487328] i2c_lock_bus: i2c-4 entry
[ 6889.487329] i2c_lock_bus: i2c-4 exit
[ 6889.487330] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.487333] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.487592] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.487595] i2c_unlock_bus: i2c-4 entry
[ 6889.487597] i2c_unlock_bus: i2c-4 exit
[ 6889.487600] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.487602] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.487606] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.487610] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.487611] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.487612] Call trace:
[ 6889.487615] dump_backtrace+0x0/0x1e0
[ 6889.487618] show_stack+0x30/0x40
[ 6889.487622] dump_stack+0xf0/0x130
[ 6889.487627] i2c_smbus_xfer+0x1ac/0x270
[ 6889.487629] i2c_smbus_read_byte_data+0x54/0x88
[ 6889.487633] nvvrs_pseq_irq_clear+0x6c/0x140
[ 6889.487635] regmap_irq_thread+0x37c/0x5c8
[ 6889.487639] irq_thread_fn+0x34/0xb0
[ 6889.487644] irq_thread+0x16c/0x2e0
[ 6889.487648] kthread+0x148/0x168
[ 6889.487650] ret_from_fork+0x10/0x18
[ 6889.487652] i2c_lock_bus: i2c-4 entry
[ 6889.487654] i2c_lock_bus: i2c-4 exit
[ 6889.487655] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.487657] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.487923] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.487926] i2c_unlock_bus: i2c-4 entry
[ 6889.487928] i2c_unlock_bus: i2c-4 exit
[ 6889.487931] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.487932] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.487934] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.487936] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.487938] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.487940] Call trace:
[ 6889.487944] dump_backtrace+0x0/0x1e0
[ 6889.487949] show_stack+0x30/0x40
[ 6889.487951] dump_stack+0xf0/0x130
[ 6889.487955] i2c_smbus_xfer+0x1ac/0x270
[ 6889.487957] i2c_smbus_read_byte_data+0x54/0x88
[ 6889.487961] nvvrs_pseq_irq_clear+0x6c/0x140
[ 6889.487966] regmap_irq_thread+0x37c/0x5c8
[ 6889.487973] irq_thread_fn+0x34/0xb0
[ 6889.487975] irq_thread+0x16c/0x2e0
[ 6889.487979] kthread+0x148/0x168
[ 6889.487981] ret_from_fork+0x10/0x18
[ 6889.487984] i2c_lock_bus: i2c-4 entry
[ 6889.487987] i2c_lock_bus: i2c-4 exit
[ 6889.487990] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.487992] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.488252] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.488256] i2c_unlock_bus: i2c-4 entry
[ 6889.488261] i2c_unlock_bus: i2c-4 exit
[ 6889.488262] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.488267] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.488268] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.488270] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.488271] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.488274] Call trace:
[ 6889.488278] dump_backtrace+0x0/0x1e0
[ 6889.488282] show_stack+0x30/0x40
[ 6889.488283] dump_stack+0xf0/0x130
[ 6889.488286] i2c_smbus_xfer+0x1ac/0x270
[ 6889.488289] i2c_smbus_read_byte_data+0x54/0x88
[ 6889.488291] nvvrs_update_bits.isra.0+0x40/0xc0
[ 6889.488294] nvvrs_rtc_disable_alarm+0x5c/0x128
[ 6889.488298] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.488299] handle_nested_irq+0xc8/0x140
[ 6889.488301] regmap_irq_thread+0x34c/0x5c8
[ 6889.488305] irq_thread_fn+0x34/0xb0
[ 6889.488307] irq_thread+0x16c/0x2e0
[ 6889.488309] kthread+0x148/0x168
[ 6889.488315] ret_from_fork+0x10/0x18
[ 6889.488316] i2c_lock_bus: i2c-4 entry
[ 6889.488318] i2c_lock_bus: i2c-4 exit
[ 6889.488320] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.488322] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.488577] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.488580] i2c_unlock_bus: i2c-4 entry
[ 6889.488585] i2c_unlock_bus: i2c-4 exit
[ 6889.488586] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.488588] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.488589] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.488591] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.488593] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.488594] Call trace:
[ 6889.488597] dump_backtrace+0x0/0x1e0
[ 6889.488602] show_stack+0x30/0x40
[ 6889.488606] dump_stack+0xf0/0x130
[ 6889.488608] i2c_smbus_xfer+0x1ac/0x270
[ 6889.488610] i2c_smbus_write_byte_data+0x58/0x80
[ 6889.488613] nvvrs_update_bits.isra.0+0x64/0xc0
[ 6889.488615] nvvrs_rtc_disable_alarm+0x5c/0x128
[ 6889.488617] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.488618] handle_nested_irq+0xc8/0x140
[ 6889.488623] regmap_irq_thread+0x34c/0x5c8
[ 6889.488628] irq_thread_fn+0x34/0xb0
[ 6889.488629] irq_thread+0x16c/0x2e0
[ 6889.488632] kthread+0x148/0x168
[ 6889.488635] ret_from_fork+0x10/0x18
[ 6889.488636] i2c_lock_bus: i2c-4 entry
[ 6889.488637] i2c_lock_bus: i2c-4 exit
[ 6889.488638] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.488642] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.488871] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.488872] i2c_unlock_bus: i2c-4 entry
[ 6889.488873] i2c_unlock_bus: i2c-4 exit
[ 6889.488874] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.488876] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.488878] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.488879] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.488881] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.488884] Call trace:
[ 6889.488889] dump_backtrace+0x0/0x1e0
[ 6889.488891] show_stack+0x30/0x40
[ 6889.488893] dump_stack+0xf0/0x130
[ 6889.488896] i2c_smbus_xfer+0x1ac/0x270
[ 6889.488899] i2c_smbus_read_byte_data+0x54/0x88
[ 6889.488900] nvvrs_update_bits.isra.0+0x40/0xc0
[ 6889.488902] nvvrs_rtc_disable_alarm+0x84/0x128
[ 6889.488908] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.488909] handle_nested_irq+0xc8/0x140
[ 6889.488913] regmap_irq_thread+0x34c/0x5c8
[ 6889.488915] irq_thread_fn+0x34/0xb0
[ 6889.488917] irq_thread+0x16c/0x2e0
[ 6889.488920] kthread+0x148/0x168
[ 6889.488922] ret_from_fork+0x10/0x18
[ 6889.488925] i2c_lock_bus: i2c-4 entry
[ 6889.488929] i2c_lock_bus: i2c-4 exit
[ 6889.488930] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.488931] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.489190] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.489191] i2c_unlock_bus: i2c-4 entry
[ 6889.489193] i2c_unlock_bus: i2c-4 exit
[ 6889.489194] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.489196] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.489197] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.489198] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.489201] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.489202] Call trace:
[ 6889.489204] dump_backtrace+0x0/0x1e0
[ 6889.489207] show_stack+0x30/0x40
[ 6889.489210] dump_stack+0xf0/0x130
[ 6889.489214] i2c_smbus_xfer+0x1ac/0x270
[ 6889.489218] i2c_smbus_write_byte_data+0x58/0x80
[ 6889.489220] nvvrs_update_bits.isra.0+0x64/0xc0
[ 6889.489221] nvvrs_rtc_disable_alarm+0x84/0x128
[ 6889.489223] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.489226] handle_nested_irq+0xc8/0x140
[ 6889.489228] regmap_irq_thread+0x34c/0x5c8
[ 6889.489231] irq_thread_fn+0x34/0xb0
[ 6889.489236] irq_thread+0x16c/0x2e0
[ 6889.489238] kthread+0x148/0x168
[ 6889.489241] ret_from_fork+0x10/0x18
[ 6889.489242] i2c_lock_bus: i2c-4 entry
[ 6889.489244] i2c_lock_bus: i2c-4 exit
[ 6889.489245] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.489246] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.489470] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.489471] i2c_unlock_bus: i2c-4 entry
[ 6889.489473] i2c_unlock_bus: i2c-4 exit
[ 6889.489474] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.489475] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.489478] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.489480] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.489483] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.489485] Call trace:
[ 6889.489488] dump_backtrace+0x0/0x1e0
[ 6889.489490] show_stack+0x30/0x40
[ 6889.489492] dump_stack+0xf0/0x130
[ 6889.489494] i2c_smbus_xfer+0x1ac/0x270
[ 6889.489496] i2c_smbus_write_byte_data+0x58/0x80
[ 6889.489499] nvvrs_rtc_update_alarm_reg+0x40/0x158
[ 6889.489504] nvvrs_rtc_disable_alarm+0xa4/0x128
[ 6889.489505] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.489507] handle_nested_irq+0xc8/0x140
[ 6889.489510] regmap_irq_thread+0x34c/0x5c8
[ 6889.489512] irq_thread_fn+0x34/0xb0
[ 6889.489514] irq_thread+0x16c/0x2e0
[ 6889.489518] kthread+0x148/0x168
[ 6889.489521] ret_from_fork+0x10/0x18
[ 6889.489525] i2c_lock_bus: i2c-4 entry
[ 6889.489526] i2c_lock_bus: i2c-4 exit
[ 6889.489527] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.489528] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.489758] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.489760] i2c_unlock_bus: i2c-4 entry
[ 6889.489762] i2c_unlock_bus: i2c-4 exit
[ 6889.489764] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.489769] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.489770] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.489771] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.489772] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.489773] Call trace:
[ 6889.489775] dump_backtrace+0x0/0x1e0
[ 6889.489779] show_stack+0x30/0x40
[ 6889.489780] dump_stack+0xf0/0x130
[ 6889.489784] i2c_smbus_xfer+0x1ac/0x270
[ 6889.489789] i2c_smbus_write_byte_data+0x58/0x80
[ 6889.489791] nvvrs_rtc_update_alarm_reg+0x60/0x158
[ 6889.489793] nvvrs_rtc_disable_alarm+0xa4/0x128
[ 6889.489796] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.489797] handle_nested_irq+0xc8/0x140
[ 6889.489799] regmap_irq_thread+0x34c/0x5c8
[ 6889.489802] irq_thread_fn+0x34/0xb0
[ 6889.489807] irq_thread+0x16c/0x2e0
[ 6889.489809] kthread+0x148/0x168
[ 6889.489812] ret_from_fork+0x10/0x18
[ 6889.489814] i2c_lock_bus: i2c-4 entry
[ 6889.489815] i2c_lock_bus: i2c-4 exit
[ 6889.489816] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.489819] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.490041] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.490042] i2c_unlock_bus: i2c-4 entry
[ 6889.490047] i2c_unlock_bus: i2c-4 exit
[ 6889.490050] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.490052] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.490054] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.490055] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.490056] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.490057] Call trace:
[ 6889.490060] dump_backtrace+0x0/0x1e0
[ 6889.490063] show_stack+0x30/0x40
[ 6889.490065] dump_stack+0xf0/0x130
[ 6889.490067] i2c_smbus_xfer+0x1ac/0x270
[ 6889.490073] i2c_smbus_write_byte_data+0x58/0x80
[ 6889.490075] nvvrs_rtc_update_alarm_reg+0x80/0x158
[ 6889.490078] nvvrs_rtc_disable_alarm+0xa4/0x128
[ 6889.490079] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.490080] handle_nested_irq+0xc8/0x140
[ 6889.490083] regmap_irq_thread+0x34c/0x5c8
[ 6889.490087] irq_thread_fn+0x34/0xb0
[ 6889.490089] irq_thread+0x16c/0x2e0
[ 6889.490095] kthread+0x148/0x168
[ 6889.490097] ret_from_fork+0x10/0x18
[ 6889.490098] i2c_lock_bus: i2c-4 entry
[ 6889.490101] i2c_lock_bus: i2c-4 exit
[ 6889.490102] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.490103] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.490325] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.490326] i2c_unlock_bus: i2c-4 entry
[ 6889.490328] i2c_unlock_bus: i2c-4 exit
[ 6889.490329] i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.490331] i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.490332] i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.490334] CPU: 0 PID: 174 Comm: irq/254-nvvrs-p Not tainted 5.10.192+ #29
[ 6889.490335] Hardware name: NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 202210.4-90bcc21 06/06/2024
[ 6889.490338] Call trace:
[ 6889.490343] dump_backtrace+0x0/0x1e0
[ 6889.490345] show_stack+0x30/0x40
[ 6889.490346] dump_stack+0xf0/0x130
[ 6889.490350] i2c_smbus_xfer+0x1ac/0x270
[ 6889.490352] i2c_smbus_write_byte_data+0x58/0x80
[ 6889.490354] nvvrs_rtc_update_alarm_reg+0xa0/0x158
[ 6889.490358] nvvrs_rtc_disable_alarm+0xa4/0x128
[ 6889.490361] nvvrs_rtc_irq_handler+0x24/0x60
[ 6889.490364] handle_nested_irq+0xc8/0x140
[ 6889.490366] regmap_irq_thread+0x34c/0x5c8
[ 6889.490368] irq_thread_fn+0x34/0xb0
[ 6889.490372] irq_thread+0x16c/0x2e0
[ 6889.490374] kthread+0x148/0x168
[ 6889.490376] ret_from_fork+0x10/0x18
[ 6889.490381] i2c_lock_bus: i2c-4 entry
[ 6889.490382] i2c_lock_bus: i2c-4 exit
[ 6889.490384] __i2c_smbus_xfer: i2c-4 entry(60)
[ 6889.490386] __i2c_smbus_xfer: i2c-4 xfer to addr 60
[ 6889.490612] __i2c_smbus_xfer: i2c-4 exit(60) with 0
[ 6889.490613] i2c_unlock_bus: i2c-4 entry
[ 6889.490614] i2c_unlock_bus: i2c-4 exit
[ 6889.490615] i2c_smbus_xfer: i2c-4 exit(60) with 0

From tegra234-platforms-vrs.dtsi:

aliases {
       rtc0 = &vrs_rtc;
       rtc1 = &tegra_rtc;
   };

   bpmp {
       i2c {
           vrs@3c {
               compatible = "nvidia,vrs-pseq";
               reg = <0x3c>;
               interrupt-parent = <&tegra_pmc>;
               /* VRS Wake ID is 24 */
               interrupts = <24 IRQ_TYPE_LEVEL_LOW>;
               interrupt-controller;
               #interrupt-cells = <2>;
               status = "okay";

               vrs_rtc: rtc {
                   status = "okay";
               };
           };
       };
   };

Do you have any idea what may cause this and how to solve this situation?

We have a custom carrier board and use a version:
# R35 (release), REVISION: 5.0, GCID: 35550185, BOARD: t186ref, EABI: aarch64, DATE: Tue Feb 20 04:46:31 UTC 2024

Hi,
If the device cannot be flashed/booted, please refer to the page to get uart log from the device:
Jetson/General debug - eLinux.org
And get logs of host PC and Jetson device for reference. If you are using custom board, you can compare uart log of developer kit and custom board to get more information.
Also please check FAQs:
Jetson AGX Orin FAQ
If possible, we would suggest follow quick start in developer guide to re-flash the system:
Quick Start — NVIDIA Jetson Linux Developer Guide 1 documentation
And see if the issue still persists on a clean-flashed system.
Thanks!

The issue here is not with flashing nor booting. In that sense, the device works fine. The problem is that i2c-adapter gets blocked by the flooding IRQs. We are trying to determine what may trigger this behavior and were hoping to get some hints about what to look for or ways to circumvent the issue.

Hi jval,

Can you apply this change and check:

— a/drivers/rtc/nvvrs-pseq-rtc.c
+++ b/drivers/rtc/nvvrs-pseq-rtc.c
@@ -142,7 +142,6 @@ static int nvvrs_rtc_update_alarm_reg(struct i2c_client *client,
}

out:
- mutex_unlock(&info->lock);
return ret;
}

Just remove mutex_unlock(&info->lock); from “nvvrs_rtc_update_alarm_reg”

Also, can you share complete dmesg logs when the issue appears ?