JetPack 4.6.3: dmesg hangs the kernel and the device reboots

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hey,

@suhash - for the NVidia staff:

I recently updated our images to from JetPack 4.5.x → 4.6.3.

When the preemp-rt patches are applied to the kernel (./rt-patch.sh apply-patches), calling dmesg will cause the kernel to lock and the device reboots after few seconds.

Using journalctl -k works fine.

I see this with TX2 and Xavier AGX devkits.

@madisox hinted at some changes in kernel/printk 32.7.2 → 32.7.3 (preempt-rt: dmesg causes device hang and reboot · Issue #1165 · OE4T/meta-tegra · GitHub). Could there be a patch missing for preempt-rt?

3

Hi,
Please check if below steps are correct:

  1. You have Xavier in Jetpack 4.5(or 4.5.1) with RT kernel image
  2. You build OTA images on Jetpack 4.6.3 with RT kernel image
  3. After upgrading the Xavier through OTA update, the system cannot boot up

Would like to make sure we understand the use-case. Please help check and confirm it.

4

The problem isn’t with OTA. I actually simply re-flashed the device for the test.

The problem is typing dmsg command from the serial console or SSH hangs the kernel (the command never returns or print anything) and the device reboots. This happens only when the preempt-rt patches have been applied to the kernel sources.

7

Hi,
Please try Jetpack 4.6(r32.6.1) instead of 4.6.3. See if it works in previous release.

9

Hi,

I have a build running to verify this with 4.6. But I cannot downgrade, I need 4.6.3 to get support for the Jetson TX2 modules with new Hynix memory. New modules are already on their way.

Are you able to reproduce the issue on your side with 4.6.3?

10

I can confirm dmesg works with Jetpack 4.6(r32.6.1).

11

@DaneLLL this turns out being more severe than originally thought. Simply stepping through the code with remote GDB makes the device hang and reboot. Recompiling the kernel without preempt rt patches makes the problem disappear, but I need the patches for our application.

I tried with both TX2 and Xavier AGX devkits.

12

Hi,
We are investigating the issue. Will update if there is further finding.

13

@DaneLLL thanks for looking into this.

Alternatively, which sw component should be changed to bring support for the new Hynix memory? cboot only? If so, would it be feasible to apply the patches only on top of JetPack 4.5.1 cboot sources?

14

Hi,
We are not able to support new modules in previous release(s). Will try to check this on latest release and update.