Jetson Xavier NX stuck in force recovery mode

[holmesc]

I believe that my Xavier Jetson NX is currently stuck in force recovery mode. I attempted to flash the device using NVIDIA SDK manager. I use the manual setup set of instructions as I have tried to use the automatic setup and it would fail.

I put the device into force recovery mode and started the flashing process. The flash completed then I was prompted to “Wait for OS login screen and complete ubuntu setup”. From prior experience, once the flash is finished the device will take itself out of force recovery mode and boot up so that I can log in. Now it doesnt do anything. The power LED is on so I know the board has power. The fan on the Jetson module starts spinning for approximately 10 seconds then shuts off. I get no signal to my display and the device doesn’t boot. Is there a way that i can manually take the device out of recovery mode? can someone please assist?

[linuxdev]

It sounds like flash actually did succeed, and it rebooted, but the GUI is not working (this is somewhat common). Can you check for a serial console boot log? Serial console survives even when the display driver or many parts of the system have crashed. Odds are that it fully booted other than the display not working. Even the fan behavior is normal and suggests that it started normally and shut off because there was insufficient heat to need the fan (which does not work if it crashes…in which case the fan either never turns on, or else turns on permanently).

[holmesc]

I tired to obtain the serial console log and fail. I have tried this over every comms channel that i can find. Whether it be pulling from the UART using and FTDI converter or connecting to a windows machine and using PuTTy. Nothing is being displayed. I am starting to think that this device might be bricked. Can you think of any other possible way i can pull information from this device?

[holmesc]

I managed to optain serial console data using PuTTy. Please see below

[0000.025] W> RATCHET: MB1 binary ratchet value 4 is larger than ratchet level 2 from HW fuses.
[0000.033] I> MB1 (prd-version: 2.3.0.0-t194-41334769-0a17edc1)
[0000.039] I> Boot-mode: Coldboot
[0000.041] I> Platform: Silicon
[0000.044] I> Chip revision : A02P
[0000.047] I> Bootrom patch version : 15 (correctly patched)
[0000.052] I> ATE fuse revision : 0x200
[0000.056] I> Ram repair fuse : 0x0
[0000.059] I> Ram Code : 0x0
[0000.062] I> rst_source: 0x0, rst_level: 0x0
[0000.066] I> Boot-device: QSPI (instance: 0)
[0000.070] I> Qspi flash params source = brbct
[0000.074] I> Qspi clock source : pllp
[0000.078] I> Qspi-0 initialized successfully
[0000.082] I> Boot chain mechanism: A/B
[0000.085] I> Current Boot-Chain Slot: 0
[0000.089] I> BR-BCT Boot-Chain: 0, status: 0. update flag: 0
[0000.094] I> Qspi flash params source = brbct
[0000.100] W> PROD_CONFIG: device prod data is empty in MB1 BCT.
[0000.107] I> Temperature = 29000
[0000.110] W> Skipping boost for clk: BPMP_CPU_NIC
[0000.115] W> Skipping boost for clk: BPMP_APB
[0000.119] W> Skipping boost for clk: AXI_CBB
[0000.123] W> Skipping boost for clk: AON_CPU_NIC
[0000.127] W> Skipping boost for clk: CAN1
[0000.131] W> Skipping boost for clk: CAN2
[0000.135] I> Boot-device: QSPI (instance: 0)
[0000.139] I> Qspi flash params source = mb1bct
[0000.143] I> Qspi clock source : pllc_out0
[0000.147] I> Qspi-0 reinitialized
[0000.150] I> Qspi flash params source = mb1bct
[0000.163] I> Non-ECC region[0]: Start:0x80000000, End:0x100000000
[0000.170] W>  Thermal config not found in BCT
[0000.178] W>  MEMIO rail config not found in BCT
[0000.188] I> Qspi flash params source = mb1bct
[0000.201] I> Qspi flash params source = mb1bct
[0000.235] I> Qspi flash params source = mb1bct
[0000.799] I> Qspi flash params source = mb1bct
[0000.816] I> Qspi flash params source = mb1bct
[0000.855] W>  Platform config not found in BCT
[0000.882] I> Qspi flash params source = mb1bct
[0000.907] I> MB1 done

▒▒▒▒main enter
SPE VERSION #: R01.00.18 Created: Jan 29 2021 @ 14:18:27
HW Function test
Start Scheduler.
in late init
▒▒
  [0000.917] I> Welcome to MB2(TBoot-BPMP) (version: default.t194-mobile-1ca012e4)
[0000.917] I> DMA Heap @ [0x526fa000 - 0x52ffa000]
[0000.918] I> Default Heap @ [0xd486400 - 0xd48a400]
[0000.919] E> DEVICE_PROD: Invalid value data = 70020000, size = 0.
[0000.924] W> device prod register failed
[0000.928] I> gpio framework initialized
[0000.932] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driver
[0000.939] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon' driver
[0000.947] I> sdcard_params [0, 55, 0, 50] in mb1_bct
[0000.952] I> Boot_device: QSPI_FLASH instance: 0
[0000.956] I> qspi flash-0 params source = boot args
[0000.962] I> QSPI-0l initialized successfully
[0000.972] I> Found 41 partitions in QSPI_FLASH (instance 0)
[0000.973] I> Active Boot chain : 0
[0001.645] I> Relocating BR-BCT
[0001.646]  > DEVICE_PROD: device prod is not initialized.
[0001.672] E> I2C: slave not found in slaves.
[0001.673] E> I2C: Could not write 0 bytes to slave: 0x00ae with repeat start true.
[0001.674] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.675] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xae at 0x00000000 via instance 0.
[0001.676] E> eeprom: Failed to read I2C slave device
[0001.679] I> Failed to read CVB eeprom data @ AE
[0001.683] I> Retrying CVB eeprom read @ AC ...
[0001.688] E> I2C: slave not found in slaves.
[0001.692] E> I2C: Could not write 0 bytes to slave: 0x00ac with repeat start true.
[0001.700] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.706] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xac at 0x00000000 via instance 0.
[0001.715] E> eeprom: Failed to read I2C slave device
[0001.719] I> Failed to read CVB eeprom data @ AC
[0001.981] I> Relocating OP-TEE dtb from: 0x6bfff1d0 to 0x70050000, size: 1008
[0001.982] I> [0] START: 0x80000000, SIZE: 0x2f000000
[0001.982] I> [1] START: 0xaf010000, SIZE: 0x18bf0000
[0001.983] I> [2] START: 0xc7d00000, SIZE: 0xc0000
[0001.983] I> [3] START: 0xca000000, SIZE: 0x800000
[0001.983] I> dram_block larger than 80000000
[0001.986] I> [4] START: 0x100000000, SIZE: 0x180000000
[0001.997] I> Setting NS memory ranges to OP-TEE dtb finished.
[0002.001] I> found decompressor handler: lz4
[0002.229] I> EKB detected (length: 0x410) @ VA:0x526ff400
[0002.231] I> Setting EKB blob info to OPTEE dtb finished.
▒▒NOTICE:  BL31: v2.6(release):4fa405dbd
NOTICE:  BL31: Built : 20:16:55, Aug 10 2022
I/TC:
▒▒
  bpmp: init
bpmp: tag is 128431eec76692047e1ac1ebc0392266
sku_dt_init: not sku 0x00
▒▒▒▒clk_early initialized
▒▒I/TC:▒▒mail_early initialized
▒▒ N▒▒fuse initialized
hwwdt initialized
▒▒on-s▒▒t194_ec_get_ec_list: found 45 ecs
▒▒ecure exter▒▒ec initialized
vmon_setup_monitors: found 3 monitors
vmon initialized
▒▒nal▒▒adc initialized
▒▒ ▒▒fmon_populate_monitors: found 73 monitors
▒▒DT found
▒▒fmon initialized
mc initialized
reset initialized
nvhs initialized
uphy_early initialized
emc_early initialized
392 clocks registered
clk initialized
io_dpd initialized
thermal initialized
thermal_mrq initialized
i2c initialized
vrmon_dt_init: vrmon node not found
vrmon_chk_boot_state: found 0 rail monitors
vrmon initialized
regulator initialized
avfs_clk_platform initialized
soctherm initialized
aotag initialized
powergate initialized
dvs initialized
pm initialized
suspend initialized
pg_late initialized
pg_mrq_init initialized
strap initialized
nvl initialized
emc initialized
emc_mrq initialized
clk_dt initialized
tj_init initialized
uphy_dt initialized
uphy_mrq initialized
uphy initialized
ec_swd_poll_start: 281 reg polling start w period 47 ms
ec_late initialized
hwwdt_late initialized
reset_mrq initialized
ec_mrq initialized
fmon_mrq initialized
clk_mrq initialized
avfs_mrq initialized
mail_mrq initialized
i2c_mrq initialized
tag_mrq initialized
console_mrq initialized
mrq initialized
clk_sync_fmon_post initialized
▒▒I/TC: OP-TEE version: 3.16 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Thu A▒▒clk_dt_late initialized
noc_late initialized
▒▒u▒▒pm_post initialized
dbells initialized
▒▒g 1▒▒dmce initialized
▒▒1▒▒cvc initialized
▒▒ 03:23:20 UT▒▒avfs_clk_mach_post initialized
▒▒C 20▒▒avfs_clk_platform_post initialized
▒▒22 ▒▒cvc_late initialized
regulator_post initialized
▒▒aarc▒▒rm initialized
console_late initialized
clk_dt_post initialized
mc_reg initialized
▒▒h64
I/TC: WARNI▒▒pg_post initialized
profile initialized
fuse_late initialized
extras_post initialentering main console loop
] ized
bpmp: init complete
▒▒NG: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Primary CPU switching to normal world boot
▒▒
  [0002.849] I> Welcome to NVDisp-Init
[0002.850] I> NVDisp-Init version: t194-f9ecfedc
[0002.850] I> CPU-BL Params @ 0xca020000
[0002.850] I>  0) Base:0x00000000 Size:0x00000000
[0002.851] I>  1) Base:0xc8300000 Size:0x00100000
[0002.851] I>  2) Base:0xc9800000 Size:0x00200000
[0002.851] I>  3) Base:0xc8600000 Size:0x00200000
[0002.854] I>  4) Base:0xc8200000 Size:0x00100000
[0002.858] I>  5) Base:0xc8100000 Size:0x00100000
[0002.863] I>  6) Base:0xc9400000 Size:0x00400000
[0002.867] I>  7) Base:0xc9000000 Size:0x00400000
[0002.872] I>  8) Base:0xc8000000 Size:0x00100000
[0002.876] I>  9) Base:0xc7f00000 Size:0x00100000
[0002.881] I> 10) Base:0xca800000 Size:0x00800000
[0002.885] I> 11) Base:0x40000000 Size:0x00040000
[0002.890] I> 12) Base:0xc7e00000 Size:0x00100000
[0002.894] I> 13) Base:0x40046000 Size:0x00002000
[0002.898] I> 14) Base:0x40048000 Size:0x00002000
[0002.903] I> 15) Base:0xaf000000 Size:0x00004000
[0002.907] I> 16) Base:0x4004a000 Size:0x00002000
[0002.912] I> 17) Base:0xc7c00000 Size:0x00100000
[0002.916] I> 18) Base:0x4004c000 Size:0x00002000
[0002.921] I> 19) Base:0xc9a00000 Size:0x00600000
[0002.925] I> 20) Base:0x4004e000 Size:0x00002000
[0002.930] I> 21) Base:0xc7dc0000 Size:0x0000c000
[0002.934] I> 22) Base:0x00000000 Size:0x00000000
[0002.939] I> 23) Base:0xc7de0000 Size:0x00020000
[0002.943] I> 24) Base:0xcc000000 Size:0x02000000
[0002.948] I> 25) Base:0x40050000 Size:0x00002000
[0002.952] I> 26) Base:0x40040000 Size:0x00006000
[0002.956] I> 27) Base:0xc8c00000 Size:0x00400000
[0002.961] I> 28) Base:0xc8400000 Size:0x00200000
[0002.965] I> 29) Base:0xc8800000 Size:0x00400000
[0002.970] I> 30) Base:0xc7dd0000 Size:0x00010000
[0002.974] I> 31) Base:0x00000000 Size:0x00000000
[0002.979] I> 32) Base:0xf8000000 Size:0x08000000
[0002.983] I> 33) Base:0xce000000 Size:0x2a000000
[0002.988] I> 34) Base:0xcb000000 Size:0x01000000
[0002.992] I> 35) Base:0xae000000 Size:0x01000000
[0002.997] I> 36) Base:0xa0000000 Size:0x0e000000
[0003.001] I> 37) Base:0xca000000 Size:0x00800000
[0003.006] I> 38) Base:0x80000000 Size:0x20000000
[0003.010] I> 39) Base:0xb0000000 Size:0x08000000
[0003.014] I> 40) Base:0x00000000 Size:0x00000000
[0003.019] I> 41) Base:0x00000000 Size:0x00000000
[0003.023] I> 42) Base:0x00000000 Size:0x00000000
[0003.028] I> 43) Base:0x00000000 Size:0x00000000
[0003.032] I> 44) Base:0x00000000 Size:0x00000000
[0003.037] I> 45) Base:0x00000000 Size:0x00000000
[0003.041] GIC-SPI Target CPU: 0
[0003.044] Interrupts Init done
[0003.047] calling constructors
[0003.050] initializing heap
[0003.053] I> Heap: [0xa0960000 ... 0xadf00000]
[0003.057] initializing threads
[0003.060] initializing timers
[0003.063] creating bootstrap completion thread
[0003.067] top of bootstrap2()
[0003.070] CPU: MIDR: 0x4E0F0040, MPIDR: 0x80000000
[0003.074] initializing platform
[0003.077] E> DEVICE_PROD: Invalid value data = 0, size = 0.
[0003.083] W> device prod register failed
[0003.086] I> Bl_dtb @0xadf00000
[0003.089] I> gpio framework initialized
[0003.101] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driver
[0003.107] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon' driver
[0003.111] I> fixed regulator driver initialized
[0003.123] I> register 'maxim' power off handle
[0003.125] I> virtual i2c enabled
[0003.125] I> registered 'maxim,max20024' pmic
[0003.126] I> tegrabl_gpio_driver_register: register 'max20024-gpio' driver
[0003.131] I> Boot-device: QSPI
[0003.134] I> Boot_device: QSPI_FLASH instance: 0
[0003.138] I> configure_qspi_clk: qparams clk_src = 5, clk_div = 0, clk_src_freq = 800000000, interface_freq = 50000000
[0003.149] I> configure_qspi_clk: FORCING INTERFACE_FREQ TO 133000000!!
[0003.156] I> QSPI source rate = 204000 Khz
[0003.159] I> Requested rate for QSPI clock = 34000 Khz
[0003.164] I> BPMP-set rate for QSPI clk = 34000 Khz
[0003.169] I> tx_clk_tap_delay : 0
[0003.172] I> rx_clk_tap_delay : 16
[0003.175] I> QSPI Flash Size = 32 MB
[0003.183] I> Qspi initialized successfully
[0003.183] I> qspi flash-0 params source = boot args
[0003.188] W> No board IDs available
[0003.191] E> Failed to get board id info!
[0003.199] I> Found 41 partitions in QSPI_FLASH (instance 0)
[0003.215] I> regulator 'vdd-hdmi-5v0' already enabled
[0003.221] I> regulator 'vdd-hdmi-5v0' already enabled
[0003.222] E> tegrabl_display_init_regulator: hdmi cable is not connected
[0003.222] E> tegrabl_display_get_pdata, failed to parse dtb settings
[0003.228] E> invalid display type
[0003.229] E> cannot find any other nvdisp nodes
[0003.230] E> no valid display unit config found in dtb
[0003.237] W> display init failed
[0003.238] initializing target
[0003.241] calling apps_init()
[0003.244] starting app kernel_boot_app
[0003.247] I> Kernel type = Normal

Jetson UEFI firmware (version 1.0-d7fb19b built on 2022-08-10T20:18:13-07:00)

ASSERT [DxeCore] /dvs/git/dirty/git-master_linux/out/nvidia/bootloader/uefi/Jets

Resetting the system in 5 seconds.
▒▒▒▒Shutdown state requested 1
Rebooting system ...
▒▒
[0000.025] W> RATCHET: MB1 binary ratchet value 4 is larger than ratchet level 2
[0000.034] I> MB1 (prd-version: 2.3.0.0-t194-41334769-0a17edc1)
[0000.039] I> Boot-mode: Coldboot
[0000.042] I> Platform: Silicon
[0000.045] I> Chip revision : A02P
[0000.048] I> Bootrom patch version : 15 (correctly patched)
[0000.053] I> ATE fuse revision : 0x200
[0000.056] I> Ram repair fuse : 0x0
[0000.059] I> Ram Code : 0x0
[0000.062] I> rst_source: 0xb, rst_level: 0x1
[0000.067] I> Boot-device: QSPI (instance: 0)
[0000.071] I> Qspi flash params source = brbct
[0000.075] I> Qspi clock source : pllp
[0000.078] I> Qspi-0 initialized successfully
[0000.082] I> Boot chain mechanism: A/B
[0000.086] I> Current Boot-Chain Slot: 1
[0000.089] I> BR-BCT Boot-Chain: 0, status: 1. update flag: 1
[0000.095] I> Qspi flash params source = brbct
[0000.101] W> PROD_CONFIG: device prod data is empty in MB1 BCT.
[0000.107] I> Temperature = 33000
[0000.110] W> Skipping boost for clk: BPMP_CPU_NIC
[0000.114] W> Skipping boost for clk: BPMP_APB
[0000.118] W> Skipping boost for clk: AXI_CBB
[0000.122] W> Skipping boost for clk: AON_CPU_NIC
[0000.126] W> Skipping boost for clk: CAN1
[0000.130] W> Skipping boost for clk: CAN2
[0000.134] I> Boot-device: QSPI (instance: 0)
[0000.138] I> Qspi flash params source = mb1bct
[0000.142] I> Qspi clock source : pllc_out0
[0000.146] I> Qspi-0 reinitialized
[0000.149] I> Qspi flash params source = mb1bct
[0000.162] I> Non-ECC region[0]: Start:0x80000000, End:0x100000000
[0000.169] W>  Thermal config not found in BCT
[0000.177] W>  MEMIO rail config not found in BCT
[0000.187] I> Qspi flash params source = mb1bct
[0000.201] I> Qspi flash params source = mb1bct
[0000.235] I> Qspi flash params source = mb1bct
[0000.799] I> Qspi flash params source = mb1bct
[0000.815] I> Qspi flash params source = mb1bct
[0000.855] W>  Platform config not found in BCT
[0000.881] I> Qspi flash params source = mb1bct
[0000.907] I> MB1 done

▒▒▒▒main enter
SPE VERSION #: R01.00.18 Created: Jan 29 2021 @ 14:18:27
HW Function test
Start Scheduler.
in late init
▒▒
  [0000.916] I> Welcome to MB2(TBoot-BPMP) (version: default.t194-mobile-1ca012e
[0000.917] I> DMA Heap @ [0x526fa000 - 0x52ffa000]
[0000.917] I> Default Heap @ [0xd486400 - 0xd48a400]
[0000.918] E> DEVICE_PROD: Invalid value data = 70020000, size = 0.
[0000.924] W> device prod register failed
[0000.928] I> gpio framework initialized
[0000.931] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driv
[0000.939] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon'
[0000.947] I> sdcard_params [0, 55, 0, 50] in mb1_bct
[0000.951] I> Boot_device: QSPI_FLASH instance: 0
[0000.956] I> qspi flash-0 params source = boot args
[0000.961] I> QSPI-0l initialized successfully
[0000.972] I> Found 41 partitions in QSPI_FLASH (instance 0)
[0000.973] I> Active Boot chain : 1
[0000.973] E> Cannot find partition bpmp-fw_b
[0000.977] E> Partition bpmp-fw_b not found
[0000.981] I> ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒: execution failed
[0000.987] I> ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒: execution failed
[0000.993] E> Top caller module: LOADER, error module: PARTITION_MANAGER, reason
[0001.002] I> AB warm reset
▒▒
[0000.025] W> RATCHET: MB1 binary ratchet value 4 is larger than ratchet level 2
[0000.033] I> MB1 (prd-version: 2.3.0.0-t194-41334769-0a17edc1)
[0000.039] I> Boot-mode: Coldboot
[0000.041] I> Platform: Silicon
[0000.044] I> Chip revision : A02P
[0000.047] I> Bootrom patch version : 15 (correctly patched)
[0000.052] I> ATE fuse revision : 0x200
[0000.056] I> Ram repair fuse : 0x0
[0000.059] I> Ram Code : 0x0
[0000.062] I> rst_source: 0xb, rst_level: 0x1
[0000.066] I> Boot-device: QSPI (instance: 0)
[0000.070] I> Qspi flash params source = brbct
[0000.074] I> Qspi clock source : pllp
[0000.078] I> Qspi-0 initialized successfully
[0000.082] I> Boot chain mechanism: A/B
[0000.085] I> Current Boot-Chain Slot: 0
[0000.089] E> Error, current slot 0 fails: sr_bl: 0x4d14ef1, sr_br: 0x0
[0000.095] I> Reset to recovery mode

[linuxdev]

I have a suggestion on the log of your post: Note that in the lower right hand corner at the end of that particular post there is a “pencil” icon. This lets you edit the post. If you add a line just above the log, and just below it, with three “single back quotes”, then it’ll add scroll bars and preserve formatting for the log. Example line looks like this:
```

Note that “`” is the unshifted tilde “~” key.

Looks like it is failing to find part of the boot software related to the boot power management processor (bpmp). Is this an SD card model? If so, then you have to flash the Jetson itself to get this updated in QSPI memory (flashing the SD card itself is not enough on those models; flashing an eMMC model would do everything needed in one flash).

You might add more info on your flash environment. In particular, is your Linux on a VM?

[holmesc]

My apologies for the incorrect posting of the log. Not on forums much.

I suppose this model would have to be an eMMC model. although i have been selecting to flash the NVMe that is installed on the jetson when prompted In the sdkmanager. Should i be selecting eMMC?

Also my linux is native 20.04

[WayneWWW]

Try to flash manually and monitor the serial console in the meantime. Attach your log as text file here.