Orin NX Custom Carrier Shutdown Implementation

Robotics & Edge Computing Jetson Systems Jetson Orin NX
, , , , , ,
1

We have noticed our custom carrier board reboots instead of shutting down when a shutdown command is issued. We want to understand why this is happening.

We see in the devkit there is a microcontroller EFM8SB10F2G or U18 that services many of the power-down request so we were wondering if there’s some logic that we’re missing that is leading to this behavior with rebooting instead of shutting down. Can you provide some explanation of what this microcontroller does when it receives a shutdown request from the SOM when SHUTDOWN_REQ is pulled low? Does it just pull POWER_EN low indefinitely and that’s how the SOM stays in a shutdown state? What would help us is if we can get a high level explanation of how the SOM stays shut-down when a shutdown command is issued and how is differs from reboot.

Something does stand out in console log when rebooting:

[ 2512.550597] systemd-shutdown[1]: Powering off.
[ 2512.550721] kvm: exiting hardware virtualization
[ 2512.604224] tegra194-pcie 140a0000.pcie: Link didn't transition to L2 state
[ 2512.724470] tegra194-pcie 140a0000.pcie: Link didn't go to detect state
[ 2513.660693] CPU:0, Error: cbb-fabric@0x13a00000, irq=22
[ 2513.660846] **************************************
[ 2513.660995] CPU:0, Error:cbb-fabric, Errmon:2
[ 2513.661125]    Error Code            : TIMEOUT_ERR
[ 2513.661236]    Overflow              : Multiple TIMEOUT_ERR
[ 2513.661376]
[ 2513.661415]    Error Code            : TIMEOUT_ERR
[ 2513.661538]    MASTER_ID             : CCPLEX
[ 2513.661636]    Address               : 0x2a080082
[ 2513.661737]    Cache                 : 0x1 -- Bufferable
[ 2513.661854]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 2513.662045]    Access_Type           : Read
[ 2513.662141]    Access_ID             : 0x14
[ 2513.662142]    Fabric                : cbb-fabric
[ 2513.662339]    Slave_Id              : 0x14
[ 2513.662426]    Burst_length          : 0x0
[ 2513.662519]    Burst_type            : 0x1
[ 2513.662609]    Beat_size             : 0x1
[ 2513.662695]    VQC                   : 0x0
[ 2513.662859]    GRPSEC                : 0x7e
[ 2513.663323]    FALCONSEC             : 0x0
[ 2513.663789]  **************************************
[ 2513.664539] ------------[ cut here ]------------
[ 2513.665234] WARNING: CPU: 0 PID: 3009 at /drivers/soc/tegra/cbb/tegra234-cbb.c:577 tegra234_cbb_isr+0x138/0x180
[ 2513.667635] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink xt_addrtype br_netfilter ip6table_nat ip6table_filter ip6_tables iptable_nat nf_nat iptable_filter ip_tables x_tables leds_max20096 userspace_alert fusb301 tegra_bpmp_thermal spi_tegra114 r8168 nv_imx264 max96793 max96792 imx283 lifmd_lvds2mipi_1 framos_common sch_fq_codel spidev pwm_fan nvgpu nvmap leds_-_uv ina3221 i2c_mux_- gpio_- fuse nfnetlink
[ 2513.706925] CPU: 0 PID: 3009 Comm: kworker/0:1 Not tainted 5.10.216-l4t-r35.6.1+ge392bfef570e #1
[ 2513.715498] Hardware name: NVIDIA - (-)/Jetson, BIOS v35.6.1 09/17/2024
[ 2513.724519] Workqueue: events_freezable pci_pme_list_scan
[ 2513.730023] pstate: 60400089 (nZCv daIf +PAN -UAO -TCO BTYPE=--)
[ 2513.735971] pc : tegra234_cbb_isr+0x138/0x180
[ 2513.740435] lr : tegra234_cbb_isr+0x114/0x180
[ 2513.744897] sp : ffff800010003bd0
[ 2513.748308] x29: ffff800010003bd0 x28: ffff2aba00ed8000
[ 2513.753821] x27: 0000000000000001 x26: 0000000000000080
[ 2513.759333] x25: ffffb6b7e9134c10 x24: ffffb6b7e99735b8
[ 2513.764847] x23: ffffb6b7e93b0008 x22: 0000000000000016
[ 2513.770360] x21: ffffb6b7e97b5500 x20: ffffb6b7e97b54f0
[ 2513.775785] x19: 0000000000000002 x18: 0000000000000006
[ 2513.781296] x17: 0000000000000000 x16: ffffb6b7e7e4f7cc
[ 2513.786722] x15: ffff8000100034e0 x14: ffff8000900036a7
[ 2513.792234] x13: ffff8000100036af x12: 0000000000000010
[ 2513.797659] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[ 2513.803172] x9 : ffffb6b7e9688890 x8 : 00000000ffffefff
[ 2513.808597] x7 : ffffb6b7e96e0890 x6 : 0000000000000000
[ 2513.814022] x5 : 0000000000000000 x4 : ffffb6b7e96e0890
[ 2513.819447] x3 : 0000000000000000 x2 : ffffb6b7e7fd13f0
[ 2513.824785] x1 : ffff2aba00ed8000 x0 : 0000000100010101
[ 2513.830123] Call trace:
[ 2513.832572]  tegra234_cbb_isr+0x138/0x180
[ 2513.836605]  __handle_irq_event_percpu+0x64/0x290
[ 2513.841322]  handle_irq_event+0x70/0x100
[ 2513.845088]  handle_fasteoi_irq+0xac/0x1f0
[ 2513.849109]  __handle_domain_irq+0x84/0xe0
[ 2513.853138]  gic_handle_irq+0xcc/0x148
[ 2513.857073]  el1_irq+0xd0/0x180
[ 2513.860048]  __do_softirq+0xac/0x3dc
[ 2513.863552]  irq_exit+0xe0/0xfc
[ 2513.866697]  __handle_domain_irq+0x8c/0xe0
[ 2513.870721]  gic_handle_irq+0xcc/0x148
[ 2513.874484]  el1_irq+0xd0/0x180
[ 2513.877641]  _raw_spin_unlock_irqrestore+0x38/0x6c
[ 2513.882452]  pci_bus_read_config_word+0x9c/0xd4
[ 2513.886998]  pci_read_config_word+0x48/0x7c
[ 2513.891022]  pci_check_pme_status+0x88/0xc0
[ 2513.895048]  pci_pme_list_scan+0x7c/0x130
[ 2513.899075]  process_one_work+0x1d8/0x4c4
[ 2513.903097]  worker_thread+0x78/0x474
[ 2513.906773]  kthread+0x158/0x15c
[ 2513.910185]  ret_from_fork+0x10/0x18
[ 2513.913860] ---[ end trace 4dcc79832e952b41 ]---
[ 2513.918455] CPU:0, Error: cbb-fabric@0x13a00000, irq=22
[ 2513.923483] **************************************
[ 2513.928471] CPU:0, Error:cbb-fabric, Errmon:2
[ 2513.932851]    Error Code            : TIMEOUT_ERR
[ 2513.936696]    Overflow              : Multiple TIMEOUT_ERR
[ 2513.941253]
[ 2513.942822]    Error Code            : TIMEOUT_ERR
[ 2513.946757]    MASTER_ID             : CCPLEX
[ 2513.949996]    Address               : 0x2a080082
[ 2513.953495]    Cache                 : 0x1 -- Bufferable
[ 2513.957697]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 2513.964521]    Access_Type           : Read
[ 2513.967759]    Access_ID             : 0x15
[ 2513.967761]    Fabric                : cbb-fabric
[ 2513.974672]    Slave_Id              : 0x14
[ 2513.977647]    Burst_length          : 0x0
[ 2513.980970]    Burst_type            : 0x1
[ 2513.984295]    Beat_size             : 0x1
[ 2513.987447]    VQC                   : 0x0
[ 2513.990333]    GRPSEC                : 0x7e
[ 2513.993483]    FALCONSEC             : 0x0
[ 2513.996459]  **************************************
[ 2514.001368] ------------[ cut here ]------------
[ 2514.006086] WARNING: CPU: 0 PID: 3009 at /drivers/soc/tegra/cbb/tegra234-cbb.c:577 tegra234_cbb_isr+0x138/0x180
[ 2514.016057] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink xt_addrtype br_netfilter ip6table_nat ip6table_filter ip6_tables iptable_nat nf_nat iptable_filter ip_tables x_tables leds_max20096 userspace_alert fusb301 tegra_bpmp_thermal spi_tegra114 r8168 nv_imx264 max96793 max96792 imx283 lifmd_lvds2mipi_1 framos_common sch_fq_codel spidev pwm_fan nvgpu nvmap leds_-_uv ina3221 i2c_mux_- gpio_- fuse nfnetlink
[ 2514.055347] CPU: 0 PID: 3009 Comm: kworker/0:1 Tainted: G        W         5.10.216-l4t-r35.6.1+ge392bfef570e #1
[ 2514.065233] Hardware name: NVIDIA - /Jetson, BIOS v35.6.1 09/17/2024
[ 2514.074249] Workqueue: events_freezable pci_pme_list_scan
[ 2514.079758] pstate: 60400089 (nZCv daIf +PAN -UAO -TCO BTYPE=--)
[ 2514.085708] pc : tegra234_cbb_isr+0x138/0x180
[ 2514.090172] lr : tegra234_cbb_isr+0x114/0x180
[ 2514.094632] sp : ffff800010003e60
[ 2514.098046] x29: ffff800010003e60 x28: ffff2aba00ed8000
[ 2514.103559] x27: 0000000000000001 x26: 0000000000000080
[ 2514.109072] x25: ffffb6b7e9134c10 x24: ffffb6b7e99735b8
[ 2514.114584] x23: ffffb6b7e93b0008 x22: 0000000000000016
[ 2514.120096] x21: ffffb6b7e97b5500 x20: ffffb6b7e97b54f0
[ 2514.125521] x19: 0000000000000002 x18: 0000000000000006
[ 2514.131034] x17: 0000000000000000 x16: ffffb6b7e7e4f7cc
[ 2514.136546] x15: ffff800010003770 x14: ffff800090003937
[ 2514.142058] x13: ffff80001000393f x12: 0000000000000010
[ 2514.147482] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[ 2514.152997] x9 : ffffb6b7e9688890 x8 : 00000000ffffefff
[ 2514.158422] x7 : ffffb6b7e96e0890 x6 : 0000000000000000
[ 2514.163847] x5 : 0000000000000000 x4 : ffffb6b7e96e0890
[ 2514.169272] x3 : 0000000000000000 x2 : ffffb6b7e7fd13f0
[ 2514.174609] x1 : ffff2aba00ed8000 x0 : 0000000100010001
[ 2514.179946] Call trace:
[ 2514.182397]  tegra234_cbb_isr+0x138/0x180
[ 2514.186425]  __handle_irq_event_percpu+0x64/0x290
[ 2514.191148]  handle_irq_event+0x70/0x100
[ 2514.194910]  handle_fasteoi_irq+0xac/0x1f0
[ 2514.198935]  __handle_domain_irq+0x84/0xe0
[ 2514.202959]  gic_handle_irq+0xcc/0x148
[ 2514.206897]  el1_irq+0xd0/0x180
[ 2514.209872]  _raw_spin_unlock_irqrestore+0x38/0x6c
[ 2514.214685]  pci_bus_write_config_word+0x78/0xb0
[ 2514.219236]  pci_write_config_word+0x48/0x70
[ 2514.223523]  pci_check_pme_status+0xb0/0xc0
[ 2514.227723]  pci_pme_list_scan+0x7c/0x130
[ 2514.231748]  process_one_work+0x1d8/0x4c4
[ 2514.235771]  worker_thread+0x78/0x474
[ 2514.239271]  kthread+0x158/0x15c
[ 2514.242685]  ret_from_fork+0x10/0x18
[ 2514.246183] ---[ end trace 4dcc79832e952b42 ]---
[ 2514.271379] tegra194-pcie 14160000.pcie: Link didn't transition to L2 state
[ 2514.333933] fusb301 11-0025: fusb301_set_mode: mode (4)(4)
[ 2514.335635] fusb301 9-0025: fusb301_set_mode: mode (4)(4)
[ 2514.350833] tegra-xusb 3610000.xhci: Firmware timestamp: 2023-02-10 03:48:10 UTC, Version: 80.05 release
[ 2514.494759] nvgpu: 17000000.ga10b                 gk20a_pm_shutdown:1325 [INFO]  shutting down
[ 2514.508132] nvgpu: 17000000.ga10b                 gk20a_pm_shutdown:1356 [INFO]  shut down complete
[ 2514.508438] arm-smmu 8000000.iommu: disabling translation
[ 2514.508732] arm-smmu 10000000.iommu: disabling translation
[ 2514.508920] arm-smmu 12000000.iommu: disabling translation
[ 2514.509205] late_shutdown started
[ 2514.509348] Disabling non-boot CPUs ...
[ 2514.538518] CPU1: shutdown
[ 2514.538603] psci: CPU1 killed (polled 0 ms)
[ 2514.562405] CPU2: shutdown
[ 2514.562494] psci: CPU2 killed (polled 0 ms)
[ 2514.586209] CPU3: shutdown
[ 2514.586300] psci: CPU3 killed (polled 0 ms)
[ 2514.5
2

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, I’ll attempt to explain the behavior you’re experiencing with your custom carrier board.

The Jetson Orin NX module has a power management system that involves a Power Sequencer or PMIC (Power Management Integrated Circuit). This system is responsible for managing the power states of the module, including shutdown and reboot.

When a shutdown request is issued, the SOM (System On Module) asserts the SHUTDOWN_REQ* signal, which is pulled low. This signal is not driven during power-on and is pulled up to VDD_IN, so it stays inactive until the module determines that the system needs to shut down.

The microcontroller on the devkit is likely responsible for servicing the power-down request and controlling the power sequencing. When the microcontroller receives a shutdown request from the SOM (SHUTDOWN_REQ* pulled low), it initiates a power-down sequence.

Here’s a high-level explanation of how the SOM stays in a shutdown state:

  1. The SOM asserts SHUTDOWN_REQ* (pulled low) to indicate that the system needs to shut down.
  2. The microcontroller on the devkit receives the shutdown request and starts the power-down sequence.
  3. The microcontroller de-asserts POWER_EN (sets it low) to disable the power supply to the module.
  4. The carrier board supplies are turned off, and the power rails are fully discharged.
  5. The SOM stays in a shutdown state, and the system is powered down.

In your case, the custom carrier board reboots instead of shutting down when a command is issued. This suggests that the power-down sequence is not being executed correctly, or the SOM is not staying in a shutdown state.

There are a few possible reasons for this behavior:

  • The microcontroller on your custom carrier board might not be servicing the power-down request correctly.
  • The power sequencing logic on your custom carrier board might be different from the devkit, leading to a reboot instead of a shutdown.
  • There might be a issue with the SHUTDOWN_REQ* signal or the POWER_EN signal, causing the SOM to reboot instead of shutting down.

To troubleshoot this issue, you can try the following:

  • Verify that the SHUTDOWN_REQ* signal is being asserted correctly when a shutdown command is issued.
  • Check the power sequencing logic on your custom carrier board to ensure it matches the devkit’s behavior.
  • Investigate the microcontroller’s role in servicing the power-down request and ensure it is functioning correctly.

Regarding the console log, it would be helpful to provide more information about the specific messages or errors you’re seeing during the reboot. This might give us a better understanding of what’s going wrong and how to fix it.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

5

Please check if the Jetson Orin NX module is affected by Jetson Orin NX Series Modules - Hardware Errata Log in | NVIDIA Developer

7

None of the errata I saw had to do with POWER_EN or shutdown or reboot.

8

Hi,

“0x2a080082” is from PCIe C8 which is the ethernet port on the Orin NX SOM.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.