firmware
t23x_general 39.2.0-gcid-45755727
The reproducer
I put together a minimal, standalone UEFI application that reproduces the trigger without any of our OS or build tooling — just a UEFI boot stick. It:
- Reads the xHCI controller’s operational registers (USBCMD, USBSTS, CONFIG, CRCR) at the fixed Tegra234 XUSB MMIO base (
0x0361_0000, the usb@3610000 node) and prints them.
- Confirms
USBCMD.RS=1 — i.e. that the controller is still the one UEFI’s own XHCI driver left running (we never reset or reprogram it).
- Behind an explicit operator keypress and a loud warning banner, issues exactly one 64-bit write to CRCR (operational-register offset
0x18): a synthetic, non-null, 64-byte-aligned pointer value with the ring-cycle-state bit set, matching the shape of the write we validated as the trigger in our own driver (see below) — not an echo of the register’s own read value, which per xHCI §5.4.5 always reads the pointer field as zero and so would write a null pointer, a different (untested) case. No doorbell is ever rung, so this value is never dereferenced; only the register write itself is exercised. Nothing else is touched.
That third step was our best hypothesis for the minimal trigger. The hardware disagreed — see the result just below. Happy to share the source on request regardless (it’s a few hundred lines of Rust against the uefi crate, no proprietary anything) — as a negative control it’s still informative.
The important result — the minimal tool does NOT reproduce the fault, and that narrows the trigger. We have now run this tool 39+ times on our affected unit with zero fault reproductions: 26+ boots in one session across two different boot sticks, and — decisively — 13 more runs minutes after our full OS path had just triggered the fault on the same board in the same power session (our kernel wall-faulted with the exact RAS record below at USB enumeration; we then booted the standalone tool repeatedly, same box, and it stayed silent every time, registers reading normally, write completing without incident). So the “maybe-the-board-just-wasn’t-in-the-faulting-state” explanation is ruled out for that window: the board demonstrably WAS in the faulting state for our full driver path, and the bare register write alone did not fire it.
What that means: a single write to this register on the untouched, UEFI-inherited, running controller is NOT sufficient to trigger the fault. Our full driver’s sequence — halt the controller, reprogram its base structures (device-context array, command ring, event ring) while halted, restart it, and only then perform the write in question as part of a slot-cleanup sequence — IS sufficient, and reproducibly so (it faulted again the same afternoon on a fresh build layout, full record below). The poisoned condition therefore appears to be reached somewhere in that fuller inherit-and-take-over sequence, not by mere register engagement. We’re keeping the tool available (it’s still a useful negative control), but the practical reproduction on your bench would need to exercise an equivalent take-over sequence against the firmware-inherited controller, not just the single write. We can share our driver’s exact sequence (it’s open) if that helps you build an internal repro.
Expected RAS record
On an affected boot we see an immediate power-off carrying this record (ARM ERR<n>STATUS layout), SNOC paired with ACI:
- Status:
0xec00030d-class
- SERR:
0xd — Illegal address (software fault)
- IERR: Carveout Uncorrectable
0x3, paired ACI SERR 0x4 / IERR FillWrite 0x9
- ADDR:
0x800000027767dc80 on our original build layout, 0x800000027767dc40 on later layouts (see “reliability” below — the low 32 bits shift by build layout, always inside the same carveout-adjacent region near the 8 GiB DRAM top)
One full raw record we captured (bit fields undecoded on our end — see the MasterID/SlaveID question below):
| field |
value |
| Status |
0xec00030d |
| MISC0 |
0x2200 |
| MISC1 |
0x0e41842000000000 |
| MISC2 / MISC3 |
0x0 / 0x4 |
| ADDR |
0x800000027767dc40 |
Within our full driver’s take-over path, it fires the instant the command-ring register machinery is engaged — the CRCR write itself is the proximate trigger there (no command or doorbell needed; a spec-legal ring re-seat write faulted 2-of-2, details in the ruled-out section below). That in-driver isolation is what motivated the standalone reproducer above — whose negative result then showed the same write, applied to the untouched inherited controller, is not sufficient on its own.
Where the fault address lands — inside an undeclared protected carveout
The fault ADDR strips (dropping the record-format high bit) to 0x27767dc80 on our original build layout, 0x27767dc40 on later layouts — a physical address near the top of DRAM. We can now say something more specific about that location than “near the carveouts,” because a separate line of investigation on the same board mapped the protected/unprotected boundaries in that region directly.
Independently of the xHCI path — during ordinary early-boot and graphics memory activity, with the USB controller not involved at all — we hit the same RAS signature (SNOC SERR=0xd Illegal-address + IERR Carveout Uncorrectable 0x3, paired ACI FillWrite 0x9) at a family of four physical addresses that climb monotonically through this region across successive boots:
0x26b900000 → 0x26bc5ee90 → 0x26c6be4a0 → 0x26d03f600
The DTB’s /reserved-memory nodes and the UEFI memory map declare carveouts at [0x26b5f0000, 0x26b7f0000) and [0x26c180000, 0x26c400000) in this range — but none of the four faulting addresses fall inside either declared window. They fall in the gaps between and above them: two below the second declared window, two above it, bracketing it. In other words a physical span running from roughly 0x26b800000 up to the framebuffer/scanout carveout at 0x279e00000 behaves as firmware-protected — a write anywhere in it raises this carveout RAS — yet nothing in the DTB or the UEFI memory map declares that span reserved. Once we exclude that entire undeclared span from our own memory map (leaving it unmapped, never written), this family of faults stops.
The xHCI fault this report is about lands inside that same undeclared-but-protected span: 0x27767dc80 (and …dc40) sits between the declared carveout top 0x26c400000 and the framebuffer carveout base 0x279e00000. So the FillWrite that powers the box off on the xHCI takeover path is not landing at an arbitrary address — it lands inside a region that independently, repeatably behaves as a protected carveout your DTB and UEFI memory map do not declare. We can’t yet prove the two are the same protection acting through one mechanism (the xHCI FillWrite reaches its target via an inherited controller pointer, not a CPU store), but they share the exact RAS signature and now, demonstrably, the same physical territory.
The question this raises for you: is the r39.2 t23x_general 39.2.0-gcid-45755727 carveout map known to under-declare protected DRAM in this range (roughly [0x26b800000, 0x279e00000), below the framebuffer carveout)? Concretely — is there firmware-side carveout/firewall protection (GSC/MC) covering physical DRAM that is not reflected in the /reserved-memory nodes or the UEFI memory map an inheriting OS sees? If so, an authoritative reserved-memory map for r39.2 — beyond what the DTB and UEFI expose — would let a downstream OS steer clear of the span entirely; today we recover it only empirically, by excluding the ground we’ve watched fault. This would also dovetail with the edk2-nvidia #111 history below (a carveout in this exact area that was, at one point, returned to the OS as usable memory).
Reliability — please read this before budgeting bench time
This is not a guaranteed-every-boot fault. It’s a per-boot “poisoned or not” coin flip, and the odds are modulated by at least two things we can point to concretely:
- Build/link layout of the software that inherits the controller. Rebuilding the exact same logical firmware/boot chain with nothing but its memory layout shifted (same code, different link addresses) changed the fault rate from 4-out-of-4 boots, to roughly 1-in-2, to 0-out-of-4 (in a 4-boot sample — see caveat just below), to 2-out-of-2 deterministic, across four distinct layouts of otherwise-identical software. Widening the sample: of six fresh layouts we built and tested that day, five sampled the fault at least once; the sixth (the 0-out-of-4 one above) did not fault in the boots we gave it, which we read as “very low probability on that specific layout in a small sample,” not “immune” — no layout we tried eliminated the fault outright when tested at a larger sample elsewhere in the same investigation. So layout clearly modulates probability (and even shifts the low bits of the fault address, as above), but does not look layout-caused; a low-rate layout should still be budgeted more than a handful of boots before concluding it’s unaffected.
- Something about the box’s state over time. Separately from this fault line, we also had a fully deterministic 2-of-2 RAS-fault reproduction (a different fault signature, in the fabric’s IOB unit, hit during our own SMP bring-up rather than the xHCI path) on one day, and the byte-identical binary, same board, booted clean 3-of-3 the next day with nothing about the binary or boot procedure changed (two of those three clean boots were on a different boot stick than the one that had faulted the day before, one was on the original — so media wasn’t held perfectly constant, but the binary and board were). For the xHCI fault line specifically, our same-session data above partially de-confounds this: within one power session the full driver path faulted while the minimal tool stayed silent 13-for-13, so at least within a session the discriminator is the SOFTWARE SEQUENCE, not drifting box state. Across days/sessions we’d still log “boots since last full power cycle” and box history alongside build layout if you reproduce this.
Practical guidance: on a unit that’s affected at all, budget for the fault to show up within the first handful of boots of the reproducer above, not necessarily the first one. If you get a long clean streak, that’s data too (matches our “some layouts are much lower-odds” observation) — it’s also worth a full power cycle (not just a warm reboot) between attempts, since we can’t rule out that state persisting across resets is part of what “poisons” a boot.
Fault signature discrimination (for context)
We tracked two distinct RAS fault lines on this board; this report and the reproducer above are about the first one only:
|
The fault this reply is about |
A separate fault line (context only) |
| Unit |
SNOC, paired with ACI |
IOB |
| Status |
0xec00030d-class |
0xe4000612 |
| SERR |
0xd — Illegal address (software fault) |
0x12 — Error response from slave |
| IERR |
Carveout Uncorrectable 0x3; ACI SERR 0x4 / IERR FillWrite 0x9 |
CBB Interface Error 0x6 |
| ADDR |
0x800000027767dc80 (or …dc40 on later layouts) |
0x8000000000000200 |
| Fires at |
first command-ring register engagement on the inherited, running xHCI controller |
early in our own multi-core bring-up, before our first core-on result |
| Effect |
RAS power-off |
RAS fault + reset loop |
The second line is mentioned only because it shares the address-encoding shape and, per the reliability notes above, may share a root-cause family with the first (firmware/fabric state that persists and drifts over time) — it is not what the reproducer targets.
What we ruled out on our own side, and why we believe this is platform/firmware-internal
Before reporting, we tried to rule out that this was something wrong in our own driver:
- Every inherited pointer we can read from DRAM is sane. We built an exhaustive census that walks every 64-bit pointer the firmware’s xHCI driver leaves behind when we take the controller over without a full hardware reset (device-context base array, per-slot device contexts, every configured endpoint’s transfer-ring dequeue pointer, the event ring, the scratchpad buffer array) and printed all of it before touching anything. Every single value is a plausible, in-range DRAM address. None of it carries the fault’s address bit pattern.
- A targeted scrub of anything that did look wrong never found anything to scrub, and the fault kept occurring on other builds in the same investigation regardless. We added a lever that would zero out any of those pointers if it ever failed a sanity check, gated to run right before the first slot-cleanup command. It no-op’d every time we tested it (nothing was ever flagged, across several boots on the build we tested it against) — and, on other build layouts tested elsewhere in the same investigation (without the scrub lever, since there was nothing for it to act on), the fault continued to occur — so whatever’s poisoned isn’t something we can see or fix from DRAM.
- The command ring’s own pointer isn’t readable to confirm or deny it as the target. Per xHCI §5.4.5, a CRCR register read always returns the Command Ring Pointer field as zero (only the status bits are live on read) — so we cannot directly inspect what pointer, if any, the controller has internally latched for its command ring. That’s part of why we built the register-level reproducer: it isolates the trigger to “engaging that specific register” without needing to know what’s behind it.
- Deliberately quiescing the command ring before touching it made the fault fire earlier, not less often. We tried reading the ring’s run state and, if it looked live, explicitly aborting and re-seating it with our own valid ring pointer before issuing any real command — a standard, spec-legal xHCI command-ring reset sequence. Every register read showed the ring already stopped (consistent with the controller being halted at that point), and the write that re-seated it — a completely ordinary CRCR write, at that point carrying a valid, non-null pointer to a ring we own — triggered the identical fault, 2-out-of-2 boots, deterministically, the instant the write posted. That let us narrow the trigger down to “a write to this register, of that general shape, while the controller is running” — which is why the standalone reproducer’s write matches that shape (a real, non-null, aligned pointer with the ring-cycle bit set) rather than testing an untried null-pointer write.
- A full controller reset (HCRST) is not usable as a workaround. We confirmed on our own hardware that issuing HCRST before taking the controller over reliably kills the firmware’s own command-processing microcontroller loop — the controller becomes permanently unable to service commands for the rest of that boot. So a full reset isn’t a viable workaround from our side, which is part of why we’re reporting this rather than working around it.
- A keyboard or any other USB device on the bus isn’t a factor. In our own driver’s automated bring-up (no operator interaction, no keyboard required) the fault reproduces with nothing on the bus but the boot media itself. (The standalone reproducer above does need a keyboard for its confirmation keypress, purely as an operator-safety gate — that’s a property of the tool, not evidence that a keyboard matters to the fault.)
Taken together, we believe the poisoned state lives in the controller/firmware itself (something retained across our controller hand-off that a full reset would clear but that we can’t clear any other way we’ve found), not in anything our driver is doing wrong or any DRAM structure we can inspect or scrub.
External references
A few things that look related from public sources:
- edk2-nvidia issue #111 (Orin NX, reported 2024, fixed in
uefi-202412.1): a UEFI commit had marked the XUSB firmware carveout as usable memory; allocations into it produced this exact signature pair — SNOC SERR 0xd Illegal-address + IERR Carveout Uncorrectable 0x3, paired ACI SERR 0x4 IERR FillWrite 0x9 — with the same 0x80000000_xxxxxxxx-prefixed fault addresses we see. This at minimum confirms the signature is this platform’s canonical “something touched the XUSB firmware carveout” record, and that carveout bookkeeping in this exact area of the UEFI tree has regressed before. One thing we could not determine from public sources: your team landed a further carveout-bookkeeping rework in uefi-202605.0 (commit 26d47b5, installing firmware-usable carveouts — explicitly including T234’s XUSB carveout — as reserved HOBs rather than returning them to the general DRAM map). That commit merged 2026-05-28 and our UEFI reports a 2026-06-01 build date, so ours may or may not contain it — the L4T release notes don’t say which tree r39.2 branched from. Could you confirm whether the r39.2 t23x_general 39.2.0-gcid-45755727 build includes that change? If it does NOT, a post-26d47b5 build would be a cheap thing for us to test if you can point us at one for this SOM; if it DOES, that would itself be informative (the fault class survives the reserved-HOB rework).
- forum topic 368712 (Orin NX, R36.5): a deliberately corrupted-but-validly-signed UEFI image crashing during its own execution produced an IOB RAS record that matches our separate fault line (mentioned above for context) field-for-field, with no custom OS or multi-core bring-up involved at all — support for treating these RAS signatures as generic “something went wrong during firmware-side execution” records rather than anything specific to what a downstream OS is doing.
- forum topic 285550 (Orin Nano 8GB SOM, L4T 35.5.0, fully stock software, no custom OS): the same IOB SERR/IERR pairing we see on our separate fault line, appearing unpredictably “after several reboots” — independent support for treating reboot count / box history, not just software correctness, as a variable in when these RAS events fire on this SoC family.
None of these are our silicon or our exact scenario, but between them they suggest this class of fault (RAS carveout/fabric events tied to inherited or firmware-owned state around the XUSB block, with occurrence correlated to build/firmware layout and boot history rather than anything a downstream OS does wrong) is a known, recurring shape on this platform rather than something specific to us.
Happy to run anything you’d like us to try on our bench, and to share the reproducer’s source — just say the word.