Is the nvvrs-pseq-rtc mutex_unlock fix in any released JetPack? IRQ "nobody cared" storm on AGX Orin (JP5.1.4 / L4T 35.6.0)

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hardware / software

  • Module: NVIDIA Jetson AGX Orin 64GB, Part Number 699-13701-0005-502 K.0
  • Carrier board: Custom ConnectTech
  • JetPack 5.1.4 / L4T 35.6.0, kernel 5.10.216-tegra, Ubuntu 20.04
  • BIOS: 202210.5-78a917ec-dirty

Summary

On a small fraction of boots (first occurrence across 100s of systems), the system comes up in a bad state:

  • timedatectl reports Failed to query server: Connection timed out
  • sudo hwclock -r -v reports either No usable clock interface found, or it opens /dev/rtc0 but then select() to /dev/rtc0 to wait for clock tick timed out

/dev/rtc0 and the /dev/rtc -> rtc0 symlink are present and correct, so this is not a missing device node.

Root cause appears to be an interrupt storm from the nvvrs-pseq PMIC

dmesg on a bad boot:

irq 244: nobody cared (try booting with the "irqpoll" option)
CPU: 0 PID: 152 Comm: irq/244-nvvrs-p Tainted: G           OE     5.10.216-tegra #1
Hardware name: NVIDIA Jetson AGX Orin/Jetson, BIOS 202210.5-78a917ec-dirty 10/10/2024
...
handle_fasteoi_irq+0xc0/0x170
...
irq_finalize_oneshot.part.0+0x70/0x110
irq_thread_fn+0x68/0xb0
irq_thread+0x164/0x2e0
handlers:
[<...>] irq_default_primary_handler threaded [<...>] regmap_irq_thread
Disabling IRQ #244

So IRQ 244 is owned by the nvvrs-pseq PMIC, and it ends with the kernel disabling the line, which matches the RTC/timedatectl failures and the system hang.

Separately, on every hwclock -r we see the following, but it appears on healthy units too, so we believe it is normal and not the fault itself:

nvvrs_pseq 4-003c: CAUTION: interrupt status reg:0x10 set to 0x8
nvvrs_pseq 4-003c: Clearing interrupts

We compared a bad unit against two known-good units: the PMIC/RTC behaviour is identical on all of them, which suggests this is a latent software issue rather than a faulty module.

What we found

These threads describe what looks like the same issue and an engineer-provided patch by removing a stray mutex_unlock(&info->lock); from nvvrs_rtc_update_alarm_reg() in drivers/rtc/nvvrs-pseq-rtc.c:

The driver is built into our kernel (=y, no loadable .ko), so we cannot determine from userspace whether our build already contains this fix (probably not though).

Questions

  1. Is this nvvrs-pseq-rtc fix included in any officially released JetPack? Or does it still exist only as the forum patch above?
  2. Does this nvvrs-pseq-rtc mutex issue match our symptoms (IRQ 244 storm and RTC/timedatectl failure)? Is it the likely root cause, or should we be looking elsewhere?
  3. Is there a recommended interim mitigation for fielded systems while we prepare a patched kernel image?

Thanks in advance.

4

Hi,
The issue is specific to AGX Orin 64GB modules. Please try the patch:
Kernel Call trace: nvvrs-pseq-irq nobody cared - #9 by WayneWWW

5

Thanks DaneLLL.

That patch is a PMIC BCT change. The other threads point to a kernel driver fix (nvvrs-pseq-rtc, the mutex_unlock line). A few quick questions:

  1. Do we need just the BCT patch, just the driver patch, or both?
  2. Is either fix already in a released JetPack (latest JP5 / JP6/ JP7)? Or do we have to patch it ourselves?
  3. I Know you said this is only on the AGX Orin 64GB (p3701-0005), but just wanted to check it dosent effect the Orin Nano 8GB and Xavier 32GB affected too?
  4. Any quick workaround for units already in the field while we build a patched image?

Thanks.

6

Hi,
The device tree patch is in r35.6.0. If the issue is still present, please apply this kernel patch and try:
Kernel thread blocking due to nvvrs-pseq-rtc IRQs - #5 by shgarg

Please follow developer guide to rebuild kernel and do image-based OTA update:

Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation
Software Packages and the Update Mechanism — NVIDIA Jetson Linux Developer Guide 1 documentation

The kernel patch is in Jetpack 6 r36 and Jetpack 7 r39.

7

Hi @DaneLLL

thats good news that the kernel patch is in Jetpack 6 r36 and Jetpack 7 r39.

Thanks for your help