Orin Nano tegra-xusb 3610000.usb: silent xHCI command-ring death under sustained USB-Ethernet load — no FW_HANG mbox, never auto-recovers

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
, ,
1

On an Orin Nano 8GB module (P3767-0000) on a custom carrier board (not the NVIDIA Orin Nano devkit), tegra-xusb 3610000.usb intermittently dies under sustained high-pps USB-Ethernet traffic. The whole USB3 bus drops at once and the controller never auto-recovers — only a
controller unbind/bind or reboot brings it back. dmesg:

Both USB-Ethernet NICs stall tx in the SAME microsecond:

[57252.271330] NETDEV WATCHDOG: enu2u4 (ax88179_178a): transmit queue 0 timed out
[57252.271330] NETDEV WATCHDOG: enu2u2 (r8152): transmit queue 0 timed out

tx-timeout recovery issues a stop-endpoint command; controller never answers:

tegra-xusb 3610000.usb: xHCI host not responding to stop endpoint command
tegra-xusb 3610000.usb: xHCI host controller not responding, assume dead
tegra-xusb 3610000.usb: HC died; cleaning up

all downstream USB devices then disconnect (root hubs persist)

(Representative of the observed sequence; the same-microsecond 57252.271330 NETDEV WATCHDOG pair is the value we have recorded. A full verbatim dmesg — including the per-CPU WARNING: backtraces and the xusb probe banner — can be captured from an affected unit on request.)

Two things this is NOT, to save us all a round-trip:

  1. Not a USB-device/dongle bug. Two different-chipset USB-Ethernet adapters (ASIX AX88179 + Realtek RTL8156B) behind this controller logged NETDEV WATCHDOG at the same microsecond (identical timestamps above), on different CPUs. The only thing the two paths share is this
    controller, so we read the NIC tx-timeouts as a downstream symptom of the controller stalling rather than a fault on either adapter. That’s what points us at the controller.
  2. Not fixed by a newer kernel/JetPack. We’re on mainline linux-yocto 6.6. We already know the JP5-era tegra_xhci_hcd_reinit() / en_hcd_reinit recovery is gone in JP6, and it keyed off the mbox FW_HANG interrupt anyway — which this crash never raises. This is the generic “assume
    dead” path, silent, no controller firmware hang log, no mbox interrupt. So no in-kernel recovery (current or back-ported) would even fire. We have an external watchdog that unbind/binds the controller and recovers it cleanly with no reboot — recovery is solved; we want the root
    cause.

It’s load-correlated and sporadic (hours to days, heavier under traffic) — looks like a race in the controller’s command/event-ring handling when stop-endpoint recovery hits an already-saturated controller, not a fixed timer.

Questions:

  1. Why does the controller enter an unrecoverable command-ring-dead state under sustained load with no FW_HANG mbox?
  2. Known errata / newer firmware for xusb FW 2023-02-10 on command-ring stalls under load?
  3. What do xHCI quirks 0x810 enable here, and are any relevant under saturation?
  4. Recommended controller-side prevention (USB3 U1/U2 LPM / ELPG off?) and tradeoffs.

Can capture and provide a full dmesg from an affected unit, and run a controlled bench repro (LPM off, pps sweep), on request. Like said earlier, it may take hours or days for the issue to come up so it would require coordinating a dedicated set up to get this if necessary.

Env: Orin Nano 8GB module (P3767-0000) on a custom carrier (not the devkit), JetPack 6 / L4T 36.x (OE4T meta-tegra), mainline linux-yocto 6.6, xusb FW 2023-02-10, xHCI quirks 0x810.

Possibly-related reports (Tegra xHCI failing/disconnecting under load, across platforms/JetPacks):

2

— 🤖 This is an automated AI response. I’m here to help, but please verify important details! —

The issue described is related to the Jetson Orin Nano’s USB controller, specifically the tegra-xusb 3610000.usb, which enters an unrecoverable command-ring-dead state under sustained high-pps USB-Ethernet traffic. This results in the whole USB3 bus dropping, and the controller never auto-recovers, requiring a controller unbind/bind or reboot to restore functionality.

Given the provided context, the problem seems to be related to a race condition in the controller’s command/event-ring handling when stop-endpoint recovery hits an already-saturated controller. The absence of a FW_HANG mbox interrupt and the lack of in-kernel recovery mechanisms (such as tegra_xhci_hcd_reinit() / en_hcd_reinit, which are gone in JetPack 6) suggest that the issue might be related to the controller’s firmware or hardware.

There are a few potential areas to explore for a solution:

  1. Firmware Update: Checking for any known errata or newer firmware for the xusb FW 2023-02-10 that might address command-ring stalls under load.
  2. xHCI Quirks: Understanding what xHCI quirks 0x810 enable and whether any are relevant under saturation conditions.
  3. Controller-Side Prevention: Exploring recommendations for preventing such issues, including settings like USB3 U1/U2 LPM (Link Power Management) or ELPG (Enhanced LPM) off, and understanding the tradeoffs.
  4. External Watchdog: Utilizing an external watchdog that can unbind/bind the controller to recover it cleanly without requiring a reboot, as mentioned, seems to be a viable workaround but does not address the root cause.

To further diagnose the issue, capturing a full dmesg from an affected unit and running a controlled bench repro (with LPM off and a pps sweep) could provide more insights. However, given the sporadic nature of the issue (occurring over hours or days under heavy traffic), coordinating a dedicated setup for this might be necessary.

In summary, the provided context suggests a complex issue related to the USB controller’s behavior under sustained load, potentially related to firmware, hardware, or specific settings. Addressing the root cause would require further investigation into the areas mentioned above.

Recommendation:

  • Investigate potential firmware updates or patches for the xusb controller.
  • Explore the impact of xHCI quirks and their settings on the controller’s behavior under load.
  • Consider controller-side settings adjustments (like LPM or ELPG) as preventive measures, weighing the tradeoffs.
  • Utilize an external watchdog as a temporary workaround for recovery, if applicable.

Given the complexity and the specificity of the issue to the Jetson Orin Nano’s USB controller behavior under sustained load, these steps aim to systematically approach the problem, focusing on firmware, settings, and potential quirks that could influence the controller’s performance and stability.

— We support team will also look into your issue to have further suggestions soon. please stay tuned. —

5

Env: Orin Nano 8GB module (P3767-0000) on a custom carrier (not the devkit), JetPack 6 / L4T 36.x (OE4T meta-tegra), mainline linux-yocto 6.6, xusb FW 2023-02-10, xHCI quirks 0x810.

Could you clarify what is the exact “x” here? Jetpack6 has been released out for many versions. Better using rel-36.5 to test.

Also, for firmware hang issue, you might need to try to reproduce issue on NV devkit. Otherwise not able to do further debug.

7

Exact version: L4T 36.4.4 (JetPack 6.2), mainline linux-yocto 6.6.

On rel-36.5: we extracted and compared the xusb_t234_prod.bin controller blob from the nvidia-l4t-xusb-firmware package across releases — it’s byte-identical, so r36.5 loads the same controller firmware and isn’t likely to change this:

┌───────────────────┬───────────────────────────┬────────┐
│ L4T release │ xusb_t234_prod.bin sha256 │ size │
├───────────────────┼───────────────────────────┼────────┤
│ 36.4.4 (ours) │ be18a8fb…44c7744 │ 156672 │
├───────────────────┼───────────────────────────┼────────┤
│ 36.5.0 (rel-36.5) │ be18a8fb…44c7744 │ 156672 │
├───────────────────┼───────────────────────────┼────────┤
│ 38.2.0 (JP7) │ be18a8fb…44c7744 │ 156672 │
└───────────────────┴───────────────────────────┴────────┘

If there’s a newer xusb FW that isn’t in the public deb feed, point us at it and we’ll test that.

One clarification that may matter for the debug path: this is not the firmware-hang case. We never see controller firmware hang and no FW_HANG mbox interrupt fires — only the generic xHCI host controller not responding, assume dead. So any recovery that keys off the FW-hang mbox
wouldn’t apply here. And the silicon clearly recovers: a plain tegra-xusb unbind/bind re-enumerates the whole bus cleanly with no reboot — so the controller isn’t physically dead, it’s stuck in a state the driver never reinitializes.

9

It actually does not matter firmware hang or not. Any kind of issue like fw hang or xhci controller died issue need to get reproduce on NV devkit first. Otherwise won’t be able to debug remotely. Some items here are not opened.