RAS Uncorrectable Error

Hi,

After several power cycles of the Jetson Orin Nano, It shows the following error:

NOTICE:  BL31: v2.8(release):e12e3fa93
NOTICE:  BL31: Built : 17:14:28, Jan  7 2025

I/TC:
I/TC: Non-secure external DT found
I/TC: OP-TEE version: 4.2 (gcc version 11.3.0 (Buildroot 2022.08)) #2 Wed Jan  8 01:24:03 UTC 2025 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Test OEM keys are being used. This is insecure for shipping products!
I/TC: fTPM ID is not enabled.
I/TC: ftpm-helper PTA: fTPM DT or EKB is not available. fTPM provisioning is not supported.
I/TC: Primary CPU switching to normal world boot
Jetson UEFI firmware (version 36.4.3-gcid-38968081 built on 2025-01-08T01:18:20+00:00)
I/TC: Reserved shared memory is disabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled

E/TC:?? 00
E/TC:?? 00 User mode data-abort at address 0x40 (translation fault)
E/TC:?? 00  esr 0x92000005  ttbr0 0x40000be291000   ttbr1 0x00000000   cidr 0x0
E/TC:?? 00  cpu #0          cpsr 0x60000000
E/TC:?? 00  x0  0000000000000000 x1  0000000000001000
E/TC:?? 00  x2  0000000000001000 x3  0000000040707bd0
E/TC:?? 00  x4  0000000040584010 x5  00000000406df568
E/TC:?? 00  x6  0000000000000040 x7  0000000000000040
E/TC:?? 00  x8  0000000000000040 x9  0000000000000000
E/TC:?? 00  x10 0000a00000001000 x11 0000000000000040
E/TC:?? 00  x12 0000000000000000 x13 4200004000000000
E/TC:?? 00  x14 0000000000000000 x15 0000000000000000
E/TC:?? 00  x16 000000004004e290 x17 00000000000000f8
E/TC:?? 00  x18 0000000000000000 x19 00000000406eb810
E/TC:?? 00  x20 0000000000000000 x21 0000000040707bd0
E/TC:?? 00  x22 000000000000003f x23 0000000040584010
E/TC:?? 00  x24 0000000000000000 x25 0000000000001000
E/TC:?? 00  x26 00000000406e8000 x27 0000000040584010
E/TC:?? 00  x28 0000000000000000 x29 0000000040707b00
E/TC:?? 00  x30 000000004060ab38 elr 00000000406df5e8
E/TC:?? 00  sp_el0 0000000040707b00
E/TC:?? 00  region  0: va 0x0000000040000000 pa 0x00000000be082000 size 0x002000 flags ---R-X
E/TC:?? 00  region  1: va 0x0000000040002000 pa 0x00000000be264000 size 0x001000 flags ---RW-
E/TC:?? 00  region  2: va 0x0000000040004000 pa 0x00000000be529000 size 0x048000 flags r-xR--
E/TC:?? 00  region  3: va 0x000000004004c000 pa 0x00000000be571000 size 0x001000 flags rw----
E/TC:?? 00  region  4: va 0x000000004004d000 pa 0x00000000be572000 size 0x00b000 flags r-x---
E/TC:?? 00  region  5: va 0x0000000040058000 pa 0x00000000be57d000 size 0x001000 flags rw----
E/TC:?? 00  region  6: va 0x0000000040059000 pa 0x00000000be57e000 size 0x001000 flags r-x---
E/TC:?? 00  region  7: va 0x000000004005a000 pa 0x00000000be57f000 size 0x2aa000 flags r-xR--
E/TC:?? 00  region  8: va 0x0000000040304000 pa 0x00000000be829000 size 0x304000 flags rw-RW-
E/TC:?? 00  region  9: va 0x0000000040608000 pa 0x00000000beb2d000 size 0x008000 flags r-x---
E/TC:?? 00  region 10: va 0x0000000040610000 pa 0x00000000beb35000 size 0x001000 flags rw-RW-
E/TC:?? 00  region 11: va 0x0000000040611000 pa 0x00000000beb36000 size 0x001000 flags r-x---
E/TC:?? 00  region 12: va 0x0000000040612000 pa 0x00000000beb37000 size 0x002000 flags rw-RW-
E/TC:?? 00  region 13: va 0x0000000040614000 pa 0x00000000beb39000 size 0x005000 flags r-x---
E/TC:?? 00  region 14: va 0x0000000040619000 pa 0x00000000beb3e000 size 0x001000 flags rw-RW-
E/TC:?? 00  region 15: va 0x000000004061a000 pa 0x00000000beb3f000 size 0x001000 flags r-x---
E/TC:?? 00  region 16: va 0x000000004061b000 pa 0x00000000beb40000 size 0x0c2000 flags rw-RW-
E/TC:?? 00  region 17: va 0x00000000406dd000 pa 0x00000000bec02000 size 0x00b000 flags r-x---
E/TC:?? 00  region 18: va 0x00000000406e8000 pa 0x00000000bec0d000 size 0x001000 flags rw-RW-
E/TC:?? 00  region 19: va 0x00000000406e9000 pa 0x00000000bec0e000 size 0x001000 flags r-x---
E/TC:?? 00  region 20: va 0x00000000406ea000 pa 0x00000000bec0f000 size 0x003000 flags rw-RW-
E/TC:?? 00  region 21: va 0x00000000406ed000 pa 0x00000000bec12000 size 0x006000 flags r-x---
E/TC:?? 00  region 22: va 0x00000000406f3000 pa 0x00000000bec18000 size 0x001000 flags rw-RW-
E/TC:?? 00  region 23: va 0x00000000406f4000 pa 0x00000000bec19000 size 0x001000 flags r-x---
E/TC:?? 00  region 24: va 0x00000000406f5000 pa 0x00000000bec1a000 size 0x002000 flags rw-RW-
E/TC:?? 00  region 25: va 0x00000000406f7000 pa 0x00000000bec1c000 size 0x005000 flags r-x---
E/TC:?? 00  region 26: va 0x00000000406fc000 pa 0x00000000bec21000 size 0x001000 flags rw-RW-
E/TC:?? 00  region 27: va 0x00000000406fd000 pa 0x00000000bec22000 size 0x001000 flags r-x---
E/TC:?? 00  region 28: va 0x00000000406fe000 pa 0x00000000bec23000 size 0x01f000 flags rw-RW-
E/TC:?? 00  region 29: va 0x000000004071d000 pa 0x00000000bec42000 size 0x015000 flags rw-RW-
E/TC:?? 00  region 30: va 0x0000000040732000 pa 0x000000000c198000 size 0x001000 flags rw----
E/TC:?? 00  region 31: va 0x0000000040733000 pa 0x0000000003270000 size 0x010000 flags rw----
E/TC:?? 00  region 32: va 0x0000000040743000 pa 0x000000000c390000 size 0x002000 flags rw----
ERROR:   Exception reason=0 syndrome=0xbe000011
ERROR:   **************************************
ERROR:   RAS Uncorrectable Error in IOB, base=0xe010000:
ERROR:          Status = 0xec000612
ERROR:   SERR = Error response from slave: 0x12
ERROR:          IERR = CBB Interface Error: 0x6
ERROR:          Overflow (there may be more errors) - Uncorrectable
ERROR:          MISC0 = 0xc4520040
ERROR:          MISC1 = 0x28cc860000000000
ERROR:          MISC2 = 0x0
ERROR:          MISC3 = 0x0
ERROR:          ADDR = 0x8000000003270000
ERROR:   **************************************
ERROR:   sdei_dispatch_event returned -1
ERROR:   **************************************

I already checked the OPTEE heap size and it is 1024 * SMALL_PAGE_SIZE, the same that it is proposed here: https://forums.developer.nvidia.com/t/orin-nx-hangs-on-optee-r35-5-0/286262/3

I noticed that someone already had this issue before with the same jetpack version (36.4.3) but nobody proposed a solution for that: Orin nano RAS Uncorrectable Error in IOB on R36.4.3 (jetpack6.2) .

Please can you help me to find the error and a reliable way to solve this in our future products using the Jetson Orin Nano?

Regards,

Andres

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with the Jetson Orin Nano, specifically the “RAS Uncorrectable Error in IOB” error, seems to be related to a hardware or firmware issue rather than a software problem.

Firstly, it’s essential to note that the OPTEE heap size you’ve set to 1024 SMALLPAGESIZE is correct and aligns with the recommended settings. However, this doesn’t seem to be the root cause of the problem.

The error message “RAS Uncorrectable Error in IOB” typically indicates a hardware failure or a fault in the Input/Output Buffer (IOB). This could be due to various reasons such as:

  1. Power cycling issues: The Jetson Orin module is not hot-pluggable, and power cycling can cause issues. Ensure that the main power supply is disconnected before installing or removing the module, and allow adequate time for the power rails to fully discharge.

  2. Voltage supply issues: Verify that the voltage supply to the VDD_IN pins is within the recommended range of 5.0V to 20V. Also, ensure that the power supply is stable and not causing any voltage fluctuations.

  3. Thermal issues: The Jetson Orin module has a shutdown temperature of 105°C. If the module is overheating, it can cause hardware failures. Ensure that the module is properly cooled, and the thermal design is adequate.

  4. Hardware faults: It’s possible that there’s a hardware fault in the IOB or other components on the module. In this case, you may need to replace the module or contact NVIDIA support for further assistance.

To troubleshoot the issue, you can try the following:

  1. Check the power supply: Verify that the power supply is stable and within the recommended range.
  2. Monitor the temperature: Use tools like tmon or jetson_stats to monitor the temperature of the module and ensure it’s within the recommended range.
  3. Run a stress test: Run a stress test on the module to see if the error occurs consistently.
  4. Try a different module: If possible, try using a different Jetson Orin Nano module to see if the issue is specific to the current module.

To prevent this issue in future products, ensure that:

  1. Proper power cycling procedures are followed: Ensure that the main power supply is disconnected before installing or removing the module, and allow adequate time for the power rails to fully discharge.
  2. Adequate cooling is provided: Ensure that the module is properly cooled, and the thermal design is adequate.
  3. Regular firmware updates are applied: Regularly update the firmware to ensure that any known issues are addressed.

In summary, the “RAS Uncorrectable Error in IOB” error on the Jetson Orin Nano is likely related to a hardware or firmware issue. By following proper power cycling procedures, ensuring adequate cooling, and monitoring the temperature, you can help prevent this issue in future products. If the problem persists, it’s recommended to contact NVIDIA support for further assistance.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

This answer is not useful because does not provide the root cause of the error neither a way to mitigate or workaround that. The ways to troubleshoot the issue have no sense because linux is not booting, not able to run 2 and 3. Power supply is ok and if I connect other jetson module it works.

The setup I’m using is the devkit, I think that the way to prevent the issue are not applicable.

It would be great a response from a human if possible.

Could yoi applied the overlay patch attached at Jetson Linux | NVIDIA Developer to see if can help.

I will try. Is there any way we can do this bootloader upgrade without modifying the other binaries already programmed in the jetson?

I don’t want to program the whole SoM and lose the way to recreate the issue (I have other SoM right now to work with, It is more important for me to understand the issue and avoid this in the final product).

Not sure whether your modules are impacted by below bulletin, suggest to apply below changes in:
Jetson Orin Series Modules Field Bulletin - Critical QSPI Software Update