URGENT! Tried to upgrade to JetPack 6.0 GA and now my Jetson is bricked

[mimzy]

I was excited this morning about upgrading my Orin AGX to 6.0 GA as I have been on the developers release and suffering through so many issues. I was excited that these issues would be gone.

I have my Jetson plugged into the network and connect to my Ubuntu 22.04 box via the USB-C ports on both devices. I can flash no problem, but after flashing, my Jetson seems to be bricked. The light is on, but no signal to the monitor and no response. Because of all the issues I have faced, I got pretty good at flashing. My process is this:

1. Shutdown the AGX using shutdown now
2. Press the Forced Recovery (middle) button with the Power On (left) button
3. Select the AGX 64 Developers Kit
4. Flash to eMMC (which has no problems)
5. After flashing, I know from experience, to give it a few minutes because the Jetson is being pre-configured and is actually downloading firmware. Sometimes I can catch it when I switch over my monitor.
6. This is where it fails with the message:

This has never happened to me before in the dozens of times I have reflashed. I can not get past this. On top of this, there is no signal from the Jetson so my monitor is blank. Clicking with mouse or tapping keys do nothing.

I have tried this several times with no difference.

On my next attempt, I decided to try and not flash with PreConfig but rather flash to Runtime as this will at least give me a live system that I can finish setting up. On this attempt, I can watch it go through the boot process then the signal to my monitor stops and the device is completely unresponsive. The boot output is scrolling by too fast to determine if there is anything up.

This is pretty awful as I have a big deadline in 4 weeks and now my Jetson seems bricked.

UPDATES:

• I tried unplugging it, and plugging it back in and sure enough, it goes through the boot process with all the messaging scrolling by. Then it gets to a point where my monitor goes blank and says no signal. I can wait, and nothing.

• Decided to try Automatic versus Manual since the SDKManager was at least recognizing the Jetson. When I did, I got the following message

Default ip is not available. Please make sure the default ip is shown in ip addr, or use a custom ip instead.

The Jetson seemed to disappear as the SDKManager said there is no device attached then the SDKManager crashed.

Also, I checked my Orbi router device list and the Jetson is not listed as a connected device. :(

NOTE: The Jetson seems to be rebooting regularly. After several minutes, the fan spins up and if I switch over to the monitor, I can see the boot log messages fly by and then it goes blank.

I can definitely confirm that the AGX is rebooting every few minutes.

[WayneWWW]

Hi,

No need to worry. I don’t think your Jetson is bricked.

Please drop sdkmanager and change to manual flash. Go to your host pc path where the sdkmanger install the BSP. The default path should be ~/nvidia/ on your host.

Go to the Linux_for_Tegra directory. Put your board into recovery mode and run this command to flash the board.

$ sudo ./flash.sh jetson-agx-orin-devkit internal

You can connect the micro usb cable to your host PC and jetson and monitoring the serial console log during the whole process.

If there is any error, share us the log you see in serial console.

NVIDIA Jetson Orin - Serial Console

I can definitely confirm that the AGX is rebooting every few minutes.

Rebooting in every few minutes could mean there are some software error but not hardware.

[suhash]

@mimzy just to confirm, you are using Jetson AGX Orin developer kit from Nvidia or using some other AGX Orin based products from our ecosystem partners ? I want to ensure that since if you are not using the developer kit and using some other carrier board, you will need to modify the BSP to adapt to your board ( you will need to contact the vendor in that case)

[mimzy]

@suhash Yes. It is an Nvidia Jetson AGX Orin 64GB Developer Kit that I purchased in January/February and it came pre-installed with Jetpack 6.0 Developers Release. It has been challenging since the beginning, but have worked around several of the issues. So much so that I have gotten very good at reflashing it, until now.

I just captured the minicom.cap file which I am uploading to a publicly accessible Google Drive.

[johnnynunez]

Disconnect everything and put only usb-C cable, keyboard etc.

[mimzy]

I was able to capture the minicom.cap file, unfortunately the forums does not seem to allow me to upload it, so you can find it here:

minicom.cap

Here are some highlights that repeat. The log captures 2 or 3 boot attempts. I eventually kill it.

RmBootstrap completed successfully
ÿá[    9.650548] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    9.656322] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[   10.142083] usb 1-4.1: Warning! Unlikely big volume range (=7248), cval->res is probably wrong.
[   10.142094] usb 1-4.1: [9] FU [PCM Playback Volume] ch = 1, val = -7264/-16/1
[   10.154077] usb 1-4.1: Warning! Unlikely big volume range (=7248), cval->res is probably wrong.
[   10.154083] usb 1-4.1: [10] FU [Mic Capture Volume] ch = 1, val = -7264/-16/1
[   10.185982] using random self ethernet address
[   10.185988] using random host ethernet address
[   12.602536] IRQ239: set affinity failed(-22).
[   12.602579] IRQ301: set affinity failed(-22).
[   12.734502] IRQ239: set affinity failed(-22).
[   12.734544] IRQ301: set affinity failed(-22).
[   12.850892] IRQ239: set affinity failed(-22).
[   12.850942] IRQ301: set affinity failed(-22).
[   12.943601] IRQ239: set affinity failed(-22).
[   12.943734] IRQ301: set affinity failed(-22).
[   24.307659] Please complete system configuration setup on desktop to proceed...
[   25.180711] CPU:0, Error: cbb-fabric@0x13a00000, irq=191
[   25.180726] **************************************
** 537 printk messages dropped **
[   25.181843]    Access_Type           : Read
** 4804 printk messages dropped **
[   25.191187]    Error Code            : TIMEOUT_ERR
** 5856 printk messages dropped **
[   25.202437]  **************************************
** 5472 printk messages dropped **
[   25.213100]    Beat_size             : 0x2
** 6447 printk messages dropped **
[   25.225904]    GRPSEC                : 0x7f
** 5481 printk messages dropped **
[   25.236842] CPU:0, Error: cbb-fabric@0x13a00000, irq=191
** 6128 printk messages dropped **
[   25.248755]    Slave_Id              : 0x3
** 5163 printk messages dropped **
[   25.258571]    Fabric                : cbb-fabric
** 5629 printk messages dropped **
[   25.268163]    Overflow              : Multiple TIMEOUT_ERR
** 4316 printk messages dropped **
[   25.275402]    Access_Type           : Read
** 5457 printk messages dropped **
[   25.284465]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
** 5262 printk messages dropped **
[   25.293127] CPU:0, Error: cbb-fabric@0x13a00000, irq=191
** 5402 printk messages dropped **
[   25.302535]    Error Code            : TIMEOUT_ERR
** 5167 printk messages dropped **
[   25.311502] **************************************
** 5914 printk messages dropped **
[   25.321442]    Access_Type           : Read
ÿèRmDeInit completed successfully
ÿá** 4498 printk messages dropped **
[   25.329025]    Burst_type            : 0x1
** 6024 printk messages dropped **
[   25.340254] CPU:0, Error:cbb-fabric, Errmon:64
** 6961 printk messages dropped **
[   25.352381]    Fabric                : cbb-fabric
** 5627 printk messages dropped **
[   25.362161]    MASTER_ID             : TSECA_NONSECURE
** 4826 printk messages dropped **
[   25.370669]    Error Code            : TIMEOUT_ERR
** 4997 printk messages dropped **
[   25.379398]  **************************************
** 4690 printk messages dropped **
[   25.387662]    Burst_type            : 0x1
** 3383 printk messages dropped **
[   25.393575]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
** 5391 printk messages dropped **
[   25.402998]
** 5788 printk messages dropped **
[   25.413049]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
** 4807 printk messages dropped **
[   25.421510]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
** 4855 printk messages dropped **

and

[   57.245550]  **************************************
[   57.245569] WARNING: CPU: 0 PID: 0 at drivers/soc/tegra/cbb/tegra234-cbb.c:608 tegra234_cbb_isr+0x14c/0x180
[   57.245947] ---[ end trace 63abfab22cc69568 ]---
[   57.447231] CPU:0, Error: sce-fabric@0xde00000, irq=187
[   57.447235] **************************************
[   57.447236] CPU:0, Error:sce-fabric, Errmon:2
[   57.447242] CBB registers returning all 1's which is invalid
[   57.447268] CPU:0, Error: cbb-fabric@0x13a00000, irq=191
[   57.447272] **************************************
[   57.447273] CPU:0, Error:cbb-fabric, Errmon:2
[   57.447280]    Error Code            : FIREWALL_ERR
[   57.447281]    Overflow              : Multiple FIREWALL_ERR
[   57.447290]
[   57.447291]    Error Code            : FIREWALL_ERR
[   57.447293]    MASTER_ID             : CCPLEX
[   57.447294]    Address               : 0xde12208
[   57.447296]    Cache                 : 0x1 -- Bufferable
[   57.447299]    Protection            : 0x2 -- Unprivileged, Non-Secure, Data Access
[   57.447301]    Access_Type           : Read
[   57.447303]    Access_ID             : 0x10
[   57.447304]    Fabric                : cbb-fabric
[   57.447306]    Slave_Id              : 0x0
[   57.447307]    Burst_length          : 0x0
[   57.447309]    Burst_type            : 0x1
[   57.447310]    Beat_size             : 0x2
[   57.447312]    VQC                   : 0x0
[   57.447314]    GRPSEC                : 0x7e
[   57.447315]    FALCONSEC             : 0x0
[   57.447317]    Slave                 : AON
[   57.447319]  **************************************
[   57.447338] WARNING: CPU: 0 PID: 0 at drivers/soc/tegra/cbb/tegra234-cbb.c:608 tegra234_cbb_isr+0x14c/0x180
[   57.447716] ---[ end trace 63abfab22cc69569 ]---
[   57.648989] CPU:0, Error: sce-fabric@0xde00000, irq=187
[   57.648992] **************************************

[mimzy]

Hopefully this will provide a solution as I am dead in the water for the prototype I am working on. Any help will be much appreciated.

[mimzy]

Thanks. I have tried pretty much everything. I am hoping the capture file will provide insights.

[WayneWWW]

Hi @mimzy

This issue is similar to the case in this post. Please take a look.

Infinite restarting after flashed Jetson Linux 6.0 production release

Sdkmanager will install the BSP directory to your host PC. The default path of this package is ~/nvidia/. This package will have flash tool which sdkmanager is using to flash your board. Thus, you can also manually flash your board by using these tools without running sdkmanager. The thing I need you to run is Following the " Skipping oem-config" in below document in your BSP directory. Flashing Support — NVIDIA Jetson Linux Developer Guide 1 documentation After running (1), flash the b…

[mimzy]

YAY! This worked. I went back to the SDKManager to do the target components which is going smoothly so far.

THANK YOU!