Jetson Xavier NX stopped working unexpectedly

Hi Team,

I have two Jetson Xavier NXs which have stopped booting up unexpectedly.
I tried accessing the debug console through minicom and I can see the following:

First NX: Prints the following and then gets stuck

[0000.024] W> RATCHET: MB1 binary ratchet value 4 is too large than ratchet level 2 from HW fuses.
[0000.033] I> MB1 (prd-version: 1.5.1.3-t194-41334769-d2a21c57)
[0000.038] I> Boot-mode: Coldboot
[0000.041] I> Chip revision : A02P
[0000.044] I> Bootrom patch version : 15 (correctly patched)
[0000.049] I> ATE fuse revision : 0x200
[0000.053] I> Ram repair fuse : 0x0
[0000.056] I> Ram Code : 0x0
[0000.059] I> rst_source : 0x0
[0000.061] I> rst_level : 0x0
[0000.065] I> Boot-device: QSPI
[0000.067] I> Qspi flash params source = brbct
[0000.072] I> Qspi using bpmp-dma
[0000.074] I> Qspi clock source : pllp
[0000.078] I> QSPI Flash Size = 32 MB
[0000.081] I> Qspi initialized successfully
[0000.085] I> Active Boot chain : 0
[0000.088] I> Boot-device: QSPI
[0000.091] I> Qspi flash params source = brbct
[0000.097] W> MB1_PLATFORM_CONFIG: device prod data is empty in MB1 BCT.
[0000.105] I> Temperature = 23000
[0000.108] W> Skipping boost for clk: BPMP_CPU_NIC
[0000.112] W> Skipping boost for clk: BPMP_APB
[0000.116] W> Skipping boost for clk: AXI_CBB
[0000.120] W> Skipping boost for clk: AON_CPU_NIC
[0000.124] W> Skipping boost for clk: CAN1
[0000.128] W> Skipping boost for clk: CAN2
[0000.132] I> Boot-device: QSPI
[0000.135] I> Boot-device: QSPI
[0000.138] I> Qspi flash params source = mb1bct
[0000.142] I> Qspi using bpmp-dma
[0000.145] I> Qspi clock source : pllc_out0
[0000.149] I> Qspi reinitialized
[0000.152] I> Qspi flash params source = mb1bct
[0000.157] I> ECC region[0]: Start:0x0, End:0x0
[0000.161] I> ECC region[1]: Start:0x0, End:0x0
[0000.166] I> ECC region[2]: Start:0x0, End:0x0
[0000.170] I> ECC region[3]: Start:0x0, End:0x0
[0000.174] I> ECC region[4]: Start:0x0, End:0x0
[0000.178] I> Non-ECC region[0]: Start:0x80000000, End:0x100000000
[0000.184] I> Non-ECC region[1]: Start:0x0, End:0x0
[0000.188] I> Non-ECC region[2]: Start:0x0, End:0x0
[0000.193] I> Non-ECC region[3]: Start:0x0, End:0x0
[0000.197] I> Non-ECC region[4]: Start:0x0, End:0x0
[0000.202] E> FAILED: Thermal config
[0000.210] E> FAILED: MEMIO rail config
[0000.220] I> Boot-device: QSPI
[0000.223] I> Qspi flash params source = mb1bct
[0000.232] I> Qspi flash params source = mb1bct
[0000.244] I> Qspi flash params source = mb1bct
[0000.310] I> Qspi flash params source = mb1bct
[0000.319] I> Qspi flash params source = mb1bct
[0000.346] I> Qspi flash params source = mb1bct
[0000.358] I> MB1 done

����main enter
SPE VERSION #: R01.00.14 Created: Sep 19 2018 @ 11:03:21
HW Function test
Start Scheduler.
in late init
��
  [0000.366] I> Welcome to MB2(TBoot-BPMP) (version: 00.00.2018.32-mobile-389501e9)
[0000.367] I> DMA Heap @ [0x526fa000 - 0x52ffa000]
[0000.367] I> Default Heap @ [0xd486400 - 0xd48a400]
[0000.368] E> DEVICE_PROD: Invalid value data = 70020000, size = 0.
[0000.374] W> device prod register failed
[0000.378] I> Boot-device: QSPI
[0000.381] I> Boot_device: QSPI_FLASH instance: 0
[0000.386] I> QSPI Flash Size = 32 MB
[0000.391] I> Qspi initialized successfully
[0000.392] I> qspi flash-0 params source = boot args
[0000.397] E> Failed: Unknown device 6
[0000.404] I> Found 47 partitions in QSPI_FLASH (instance 0)
[0000.407] I> Active Boot chain : 0
[0000.410] I> parsing oem signed section of bpmp-fw header done
[0000.415] I> bpmp-fw binary init read from storage
[0000.420] I> oem authentication of bpmp-fw header done
[0000.434] I> bpmp-fw binary done read from storage
[0000.435] I> bpmp-fw: Authentication init Done
[0000.436] I> parsing oem signed section of cpubl header done
[0000.439] I> cpubl binary init read from storage
[0000.444] I> bpmp-fw: Authentication Finalize Done
[0000.449] I> oem authentication of cpubl header done
[0000.456] I> cpubl binary done read from storage
[0000.458] I> cpubl: Authentication init Done
[0000.462] I> parsing oem signed section of rce header done
[0000.467] I> rce binary init read from storage
[0000.472] I> Relocating BR-BCT
[0000.474] I> cpubl: Authentication Finalize Done
[0000.479] I> oem authentication of rce header done
[0000.484] I> rce binary done read from storage
[0000.488] I> rce: Authentication init Done
[0000.492] I> parsing oem signed section of ape header done
[0000.497] I> ape binary init read from storage
[0000.501] I> rce: Authentication Finalize Done
[0000.506] I> oem authentication of ape header done
[0000.510] I> ape binary done read from storage
[0000.514] I> ape: Authentication init Done
[0000.519] I> parsing oem signed section of tos header done
[0000.524] I> tos binary init read from storage
[0000.528] I> ape: Authentication Finalize Done
[0000.533] I> oem authentication of tos header done
[0000.539] I> tos binary done read from storage
[0000.541] I> tos: Authentication init Done
[0000.546] I> parsing oem signed section of bpmp-fw-dtb header done
[0000.551] I> bpmp-fw-dtb binary init read from storage
[0000.556] I> tos: Authentication Finalize Done
[0000.562] I> oem authentication of bpmp-fw-dtb header done
[0000.566] I> bpmp-fw-dtb binary done read from storage
[0000.571] I> bpmp-fw-dtb: Authentication init Done
[0000.576] I> parsing oem signed section of cpubl-dtb header done
[0000.581] I> cpubl-dtb binary init read from storage
[0000.586] I> bpmp-fw-dtb: Authentication Finalize Done
[0000.642] I> oem authentication of cpubl-dtb header done
[0000.643] I> cpubl-dtb binary done read from storage
[0000.643] I> cpubl-dtb: Authentication init Done
[0000.644] I> parsing oem signed section of eks header done
[0000.645] I> eks binary init read from storage
[0000.645] I> cpubl-dtb: Authentication Finalize Done
[0000.646] I> oem authentication of eks header done
[0000.651] I> eks binary done read from storage
[0000.655] I> eks: Authentication init Done
[0000.659] I> eks: Authentication Finalize Done
[0000.663] I> EKB detected (length: 0x410) @ VA:0x52705400
��NOTICE:  BL31: v1.3(release):de895fd9e
NOTICE:  BL31: Built : 20:51:20, Oct 27 2020
ipc-unittest-main: 1519: Welcome to IPC unittest!!!
ipc-unittest-main: 1531: waiting forever
ipc-unittest-srv: 329: Init unittest services!!!
hwkey-agent: 40: hwkey-agent is running!!
hwkey-agent: 182: key_mgnt_processing .......
hwkey-agent: 157: Init hweky-agent services!!
platform_bootstrap_epilog: trusty bootstrap complete
��

welcome to lk
calling constructors
initializing heap
creating bootstrap completion thread
top of bootstrap2()
initializing platform
bpmp: platform_init
tag is 57f8a77779f848bf2ecf21dabee5645f
tag_show initialized
dt initialized
mail initialized
chipid initialized
fuse initialized
sku initialized
speedo initialized
ec_get_ec_list: found 45 ecs
ec initialized
ec_mrq initialized
vmon_populate_monitors: found 3 monitors
vmon initialized
adc initialized
fmon_populate_monitors: found 73 monitors
fmon initialized
fmon_mrq initialized
reset initialized
nvhs initialized
391 clocks registered
clk_mrq_init: mrq handler registered
clk initialized
nvlink initialized
io_dpd initialized
io_dpd initialized
thermal initialized
i2c5 controller initialized
initialized i2c mrq handling
i2c initialized
regulator initialized
avfs_clk_platform_init: bad clk id in clock@cluster1_avfs
avfs_clk_platform initialized
soctherm initialized
aotag initialized
powergate initialized
dvs initialized
pm initialized
pg_late initialized
strap initialized
tag initialized
emc initialized
clk_dt initialized
avfs_ccplex_platform initialized
tj_max: dt node not found
tj_init initialized
uphy_mrq_init: mrq handler registered
uphy_dt initialized
uphy initialized
safereg_init: period 80 ms
ec_late initialized
��
  [0001.004] I> Welcome to Cboot
[0001.005] I> Cboot Version: t194-aad9d75e
��mrq initialized
WARNING: no registered clock for FMON_NAFLL_CLUSTER1 (id 281)
fmon_post initialized
��[0001.005] I> CPU-BL Params @ 0xf2820000
[0001.010] I>  0) Base:0x00000000 Size:0x00000000
[0001.015] I>  1) Base:0xf1100000 Size:0x00100000
��clk_set_parent failed for clk can1, parent pll_aon (-22)
clk_set_parent failed for clk can2, parent pll_aon (-22)
clk_set_parent failed for clk dmic5, parent pll_aon (-22)
clk_set_parent failed for clk i2c2, parent pll_aon (-22)
clk_set_parent failed for clk i2c8, parent pll_aon (-22)
clk_set_parent failed for clk spi2, parent pll_aon (-22)
clk_set_parent failed for clk pwm4, parent pll_aon (-22)
clk_dt_late initialized
machine_check initialized
pm_post initialized
dbells initialized
avfs_clk_platform_post initialized
dmce initialized
cvc initialized
ccplex_avfs_hw_init: nafll_cluster0: not monitored
ccplex_avfs_hw_init: nafll_cluster2: not monitored
ccplex_avfs_hw_init: nafll_cluster3: not monitored
avfs_clk_mach_post initialized
regulator_post initialized
rm initialized
sc7_diag initialized
thermal_test initialized
serial_late initialized
clk_post initialized
clk_dt_post initialized
mc_reg initialized
pg_post initialized
dyn_modules initialized
sku_debugfs initialized
speedo_debugfs initialized
adc_debugfs initialized
Failed to register PTO counter for id 281
Failed to register PTO counter for id 281
Failed to register PTO counter for id 281
Failed to register PTO counter for id 281
clk_debugfs initialized
emc_debugfs initialized
dvs_debugfs initialized
fmon_debugfs_init_one: no clock debugfs node to attach FMON_NAFLL_CLUSTER1
fmon_debugfs initialized
vmon_debugfs initialized
pg_debugfs initialized
profile_fs initialized
debugfs_cons initialized
mail_fs initialized
profile initialized
cvc_debugfs initialized
dmce_debugfs initialized
ec_debugfs initialized
rm_rail_debugfs_init: /rm/vdd_cpu: failed
rm_rail_debugfs_init: /rm/vdd_cpu: failed
rm_debugfs initialized
soctherm_debug initialized
gr_reader initialized
mods initialized
dt_fs initialized
debugfs_mrq initialized
debug_mrq initialized
debug_safereg initialized
initializing target
calling apps_init()
starting app shell
entering main console loop
] ��[0001.019] I>  2) Base:0xf2000000 Size:0x00200000
[0001.199] I>  3) Base:0xf1200000 Size:0x00200000
[0001.203] I>  4) Base:0xf1000000 Size:0x00100000
[0001.208] I>  5) Base:0xf0f00000 Size:0x00100000
[0001.212] I>  6) Base:0xf3800000 Size:0x00400000
[0001.217] I>  7) Base:0xf1c00000 Size:0x00400000
[0001.221] I>  8) Base:0xf0e00000 Size:0x00100000
[0001.226] I>  9) Base:0xf0d00000 Size:0x00100000
[0001.230] I> 10) Base:0xf3000000 Size:0x00800000
[0001.235] I> 11) Base:0x40000000 Size:0x00040000
[0001.239] I> 12) Base:0xf0c00000 Size:0x00100000
[0001.243] I> 13) Base:0x40046000 Size:0x00002000
[0001.248] I> 14) Base:0x40048000 Size:0x00002000
[0001.252] I> 15) Base:0xac000000 Size:0x00004000
[0001.257] I> 16) Base:0x4004a000 Size:0x00002000
[0001.261] I> 17) Base:0xf0b00000 Size:0x00100000
[0001.266] I> 18) Base:0x4004c000 Size:0x00002000
[0001.270] I> 19) Base:0xf2200000 Size:0x00600000
[0001.275] I> 20) Base:0x4004e000 Size:0x00002000
[0001.279] I> 21) Base:0xf09d0000 Size:0x0000c000
[0001.284] I> 22) Base:0x00000000 Size:0x00000000
[0001.288] I> 23) Base:0xf09e0000 Size:0x00020000
[0001.293] I> 24) Base:0xf6000000 Size:0x02000000
[0001.297] I> 25) Base:0x40050000 Size:0x00002000
[0001.301] I> 26) Base:0x40040000 Size:0x00006000
[0001.306] I> 27) Base:0xf1800000 Size:0x00400000
[0001.310] I> 28) Base:0xf4c00000 Size:0x01400000
[0001.315] I> 29) Base:0xf1400000 Size:0x00400000
[0001.319] I> 30) Base:0xf0a00000 Size:0x00100000
[0001.324] I> 31) Base:0x00000000 Size:0x00000000
[0001.328] I> 32) Base:0xf8000000 Size:0x08000000
[0001.333] I> 33) Base:0x00000000 Size:0x00000000
[0001.337] I> 34) Base:0xf3c00000 Size:0x01000000
[0001.342] I> 35) Base:0xab000000 Size:0x01000000
[0001.346] I> 36) Base:0xa0000000 Size:0x0b000000
[0001.351] I> 37) Base:0xf2800000 Size:0x00800000
[0001.355] I> 38) Base:0x80000000 Size:0x20000000
[0001.359] I> 39) Base:0xb0000000 Size:0x08000000
[0001.364] I> 40) Base:0x00000000 Size:0x00000000
[0001.368] I> 41) Base:0x00000000 Size:0x00000000
[0001.373] I> 42) Base:0x00000000 Size:0x00000000
[0001.377] I> 43) Base:0x00000000 Size:0x00000000
[0001.382] I> 44) Base:0x00000000 Size:0x00000000
[0001.386] I> 45) Base:0x00000000 Size:0x00000000
[0001.391] GIC-SPI Target CPU: 0
[0001.394] Interrupts Init done
[0001.397] calling constructors
[0001.399] initializing heap
[0001.402] I> Heap: [0xa0691568 ... 0xab000000]
[0001.406] initializing threads
[0001.409] initializing timers
[0001.412] creating bootstrap completion thread
[0001.416] top of bootstrap2()
[0001.419] CPU: MIDR: 0x4E0F0040, MPIDR: 0x80000000
[0001.424] initializing platform
[0001.427] E> DEVICE_PROD: Invalid value data = 0, size = 0.
[0001.432] W> device prod register failed
[0001.436] I> Bl_dtb @0xaaf00000
[0001.441] W> "plugin-manager" doesn't exist, creating
[0001.444] W> "ids" doesn't exist, creating
[0001.448] W> "connection" doesn't exist, creating
[0001.452] W> "configs" doesn't exist, creating
[0001.460] I> Find /i2c@3160000's alias i2c0
[0001.461] I> Reading eeprom i2c=0 address=0x50
[0001.490] I> Device at /i2c@3160000:0x50
[0001.491] I> Reading eeprom i2c=0 address=0x57
[0001.515] I> Device at /i2c@3160000:0x57
[0001.517] I> Find /i2c@c240000's alias i2c1
[0001.517] I> Reading eeprom i2c=1 address=0x50
[0001.519] E> I2C: slave not found in slaves.
[0001.519] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[0001.520] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.521] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via insta.
[0001.529] E> eeprom: Retry to read I2C slave device.
[0001.534] E> I2C: slave not found in slaves.
[0001.538] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[0001.546] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.551] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via insta.
[0001.561] E> eeprom: Failed to read I2C slave device
[0001.566] I> Eeprom read failed 0x3526070d
[0001.570] I> create_pm_ids: id: 3668-0000-200-H, len: 15
[0001.575] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,disp3
[0001.586] I> create_pm_ids: id: 3509-0000-100-G, len: 15
[0001.591] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,disp3
[0001.602] I> Adding plugin-manager/ids/3668-0000-200=/i2c@3160000:module@0x50
[0001.609] W> "i2c@3160000" doesn't exist, creating
[0001.614] W> "module@0x50" doesn't exist, creating
[0001.618] I> Adding plugin-manager/ids/3509-0000-100=/i2c@3160000:module@0x57
[0001.625] W> "module@0x57" doesn't exist, creating
[0001.632] I> Adding plugin-manager/cvm
[0001.634] W> "chip-id" doesn't exist, creating
[0001.638] I> Adding plugin-manager/chip-id/A02P
[0001.642] I> Plugin-manager override starting
[0001.647] I> node /plugin-manager/fragment-pcie-c5-rp matches
[0001.657] I> node /plugin-manager/fragement-tegra-wdt-en matches
[0001.662] I> node /plugin-manager/fragement-tegra-sdhci-emmc-dis matches
[0001.670] I> Disable plugin-manager status in FDT
[0001.670] I> Plugin-manager override finished successfully
[0001.674] I> gpio framework initialized
[0001.679] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driver
[0001.686] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon' driver
[0001.693] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x46
[0001.700] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[0001.708] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[0001.715] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[0001.721] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x44
[0001.728] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[0001.735] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[0001.743] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[0001.749] I> fixed regulator driver initialized
[0001.757] I> register 'maxim' power off handle
[0001.758] I> virtual i2c enabled
[0001.760] I> registered 'maxim,max20024' pmic
[0001.765] I> tegrabl_gpio_driver_register: register 'max20024-gpio' driver
[0001.771] I> Boot-device: QSPI
[0001.774] I> Boot_device: QSPI_FLASH instance: 0
[0001.779] I> QSPI source rate = 204000 Khz
[0001.783] I> Requested rate for QSPI clock = 34000 Khz
[0001.788] I> BPMP-set rate for QSPI clk = 34000 Khz
[0001.793] I> QSPI Flash Size = 32 MB
[0001.800] I> Qspi initialized successfully
[0001.801] I> qspi flash-0 params source = boot args
[0001.804] I> create_pm_ids: id: 3668-0000-200-H, len: 15
[0001.810] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,disp3
[0001.821] I> create_pm_ids: id: 3509-0000-100-G, len: 15
[0001.826] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,disp3
[0001.837] I> Found sdcard
[0001.841] I> enabling 'vdd-sdmmc1-sw' regulator
[0001.847] I> regulator 'vdd-sdmmc1-sw' already enabled
[0001.853] E> Error in command_complete c8001 int_status
[0001.856] E> Error in command_complete c8001 int_status
[0001.861] E> Error in command_complete 18001 int_status
[0001.864] E> Sending CMD_SD_SEND_IF_COND failed
[0001.878] W> Error opening sdcard-0
[0001.878] I> -0 params source = 
[0001.879] E> Failed to initialize device 6-0
[0001.879] E> Error config_storage
[0001.882] initializing target
[0001.885] calling apps_init()
[0001.887] starting app kernel_boot_app
[0001.891] E> Partition manager might not be initialized.
[0001.896] E> Failed to open partition[0001.900] E> tegrabl_load_bmp_blob: BMP blob initialization fad
[0001.906] W> Loading bmp blob to memory failed
[0001.910] I> Kernel type = Normal
[0001.913] I> Loading kernel-bootctrl from partition
[0001.918] E> Partition manager might not be initialized.
[0001.923] E> Cannot open partition kernel-bootctrl
[0001.928] W> tegrabl_get_kernel_bootctrl: failed to read primary bootctrl data
[0001.935] I> Loading kernel-bootctrl_b from partition
[0001.939] E> Partition manager might not be initialized.
[0001.945] E> Cannot open partition kernel-bootctrl_b
[0001.949] W> tegrabl_get_kernel_bootctrl: failed to read recovery bootctrl data
[0001.957] W> tegrabl_get_kernel_bootctrl: use default dummy boot control data
[0001.964] 
[0001.965] -----------------------------------------------
[0001.970] Synchronous Exception: DATA ABORT (FAR: 0)
[0001.975] -----------------------------------------------
[0001.980] PAR_ELX: 0x80f
[0001.982] 
[0001.984] ESR 0x96000007: ec 0x25, il 0x1, iss 0x7
[0001.988] -----------------------------------------------
[0001.993]  [Stack Trace]
[0001.996] 
[0001.997] => pc:0xA063FB44, sp:0xA0697380
[0002.001] => pc:0xA060F450, sp:0xA06975C0
[0002.005] => pc:0xA060F5EC, sp:0xA0697660
[0002.009] => pc:0xA060F600, sp:0xA06976D0
[0002.012] => pc:0xA060F284, sp:0xA0697710
[0002.016] => pc:0xA060E7F8, sp:0xA0697720
[0002.020] => pc:0xA060E7CC, sp:0xA0697730
[0002.024] -----------------------------------------------
[0002.029] iframe 0xa0697290:
[0002.032] x0  0x               0 x1  0x        a06976a0 x2  0x        a06976a8 x3  0x        a06976b0
[0002.041] x4  0x               0 x5  0x               0 x6  0x               0 x7  0x               0
[0002.050] x8  0x              43 x9  0x               a x10 0x        a0697748 x11 0x            88a0
[0002.059] x12 0x        a0691160 x13 0x               0 x14 0x               0 x15 0x               0
[0002.068] x16 0x               0 x17 0x               0 x18 0x               0 x19 0x               0
[0002.077] x20 0x        a06976a8 x21 0x               0 x22 0x        a066103f x23 0x        a06976a0
[0002.086] x24 0x        a06976b0 x25 0x        a06976e0 x26 0x               0 x27 0x               0
[0002.096] x28 0x               0 x29 0x        a06975c0 lr  0x        a063fb34 sp  0x        a0697380
[0002.105] elr 0x        a063fb44
[0002.108] spsr 0x        20000209
[0002.111] -----------------------------------------------
[0002.116] panic (caller 0xa0601238): die
[0002.120] HALT: spinning forever...

Second NX: Prints the following in a loop:

Is it possible to get these working again?
What can I do for the same…?

Regards,
Ritvik

It looks like the QSPI-NOR flash got corrupted in both cases. You’re probably going to need to reflash both devices.
Are you using the production EMMc version or the DevKit version of the NX?
Which Linux for Tegra/Jetpack version are you using?

Hi @gtj

I am using the production release Jetpack 4.4 (L4T 32.4.3).
I will reflash the devices today and check if it works.

Regards,
Ritvik

Hi @gtj

I tried the method described here: Getting Started With Jetson Xavier NX Developer Kit | NVIDIA Developer
to create new memory cards for both the NX.
The NXs still do not boot up.
I can see no change in messages over the debug console.

Should I try flashing with the NVIDIA SDK Manager too?

Regards,
Ritvik

The SDK manager isn’t going to help. It runs the same commands. When you reflashed, you used the same L4T version you had (32.4.3)? Can you try with L4T 32.5.1?

You should still try to reflash the board with sdkmanager.

Sdkmanager will update the bootloader in the QSPI-NOR. But sdcard image will not.

cool, will try reflashing with the sdk manager and check if it helps

Hi @WayneWWW

I was able to get the second NX working again by flashing thru SDK manager.
But the other one is not getting flashed through the SDK manager either.
The flashing gets stuck at 99%.
Below are the logs during the flashing process.

09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.1960 ] tegrarcm_v2 --isapplet
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.2165 ] tegrarcm_v2 --download bct_bootrom br_bct_BR.bct --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt --download bct_mem mem_rcm_sigheader.bct.encrypt
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.2409 ] Sending bct_mb1
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.2452 ] [...] 100%
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.2490 ] Sending bct_mem
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.2985 ] [...] 100%
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.3745 ] tegrahost_v2 --chip 0x19 --align blob_nvtboot_recovery_cpu_t194.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.3801 ] adding BCH for blob_nvtboot_recovery_cpu_t194.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.3891 ] tegrasign_v2 --key None --list blob_nvtboot_recovery_cpu_t194_sigheader.bin_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.3899 ] Assuming zero filled SBK key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.3927 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_nvtboot_recovery_cpu_t194_sigheader.bin.encrypt blob_nvtboot_recovery_cpu_t194_sigheader.bin.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4033 ] adding BCH for blob_nvtboot_recovery_t194.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4096 ] tegrasign_v2 --key None --list blob_nvtboot_recovery_t194_sigheader.bin_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4131 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_nvtboot_recovery_t194_sigheader.bin.encrypt blob_nvtboot_recovery_t194_sigheader.bin.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4183 ] tegrahost_v2 --chip 0x19 --align blob_preboot_c10_prod_cr.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4205 ] tegrahost_v2 --chip 0x19 0 --magicid MTSP --appendsigheader blob_preboot_c10_prod_cr.bin zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4266 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_preboot_c10_prod_cr_sigheader.bin.encrypt blob_preboot_c10_prod_cr_sigheader.bin.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4335 ] Header already present for blob_mce_c10_prod_cr.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4390 ] Assuming zero filled SBK key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4476 ] tegrahost_v2 --chip 0x19 --align blob_mts_c10_prod_cr.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.4514 ] adding BCH for blob_mts_c10_prod_cr.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.5603 ] tegrasign_v2 --key None --list blob_mts_c10_prod_cr_sigheader.bin_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.5615 ] Assuming zero filled SBK key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.5685 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_mts_c10_prod_cr_sigheader.bin.encrypt blob_mts_c10_prod_cr_sigheader.bin.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6237 ] adding BCH for blob_bpmp_t194.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6496 ] tegrasign_v2 --key None --list blob_bpmp_t194_sigheader.bin_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6505 ] Assuming zero filled SBK key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6547 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_bpmp_t194_sigheader.bin.encrypt blob_bpmp_t194_sigheader.bin.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6789 ] adding BCH for blob_tegra194-a02-bpmp-p3668-a00.dtb
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6854 ] tegrasign_v2 --key None --list blob_tegra194-a02-bpmp-p3668-a00_sigheader.dtb_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6886 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_tegra194-a02-bpmp-p3668-a00_sigheader.dtb.encrypt blob_tegra194-a02-bpmp-p3668-a00_sigheader.dtb.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.6987 ] adding BCH for blob_spe_t194.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7051 ] Assuming zero filled SBK key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7071 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_spe_t194_sigheader.bin.encrypt blob_spe_t194_sigheader.bin.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7165 ] adding BCH for blob_tos-trusty_t194.img
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7300 ] Assuming zero filled SBK key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7328 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_tos-trusty_t194_sigheader.img.encrypt blob_tos-trusty_t194_sigheader.img.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7457 ] tegrahost_v2 --chip 0x19 0 --magicid EKSB --appendsigheader blob_eks.img zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7484 ] tegrasign_v2 --key None --list blob_eks_sigheader.img_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7586 ] adding BCH for blob_tegra194-p3668-all-p3509-0000.dtb
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7664 ] tegrasign_v2 --key None --list blob_tegra194-p3668-all-p3509-0000_sigheader.dtb_list.xml --pubkeyhash pub_key.key
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7710 ] tegrahost_v2 --chip 0x19 0 --updatesigheader blob_tegra194-p3668-all-p3509-0000_sigheader.dtb.encrypt blob_tegra194-p3668-all-p3509-0000_sigheader.dtb.hash zerosbk
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7817 ] blobsize is 5689928
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7850 ] Added binary blob_nvtboot_recovery_t194_sigheader.bin.encrypt of size 129936
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7916 ] Added binary blob_mts_c10_prod_cr_sigheader.bin.encrypt of size 3430800
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.7962 ] tegrarcm_v2 --download blob blob.bin
09:50:19 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.8157 ] Sending blob
09:50:20 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 7.8159 ] [... ] 018% [ 7.8159 ] [... ] 036% [ 7.8159 ] [... ] 055% [ 7.8159 ] [... ] 073% [ 7.8159 ] [... ] 092% [ 7.8159 ] [...] 100%
09:50:20 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 8.5893 ] Applet version 01.00.0000
10:07:13 INFO: Flash Jetson Xavier NX (Devkit) - flash: [ 9.5991 ] tegrarcm_v2 --isapplet

Are you able to dump the flash log from uart during the flash process?

The log you just shared is the log from host. Flash process requires both side (host/NX) running some binaries. If something gets stuck, it means both sides get stuck too. Need the device side log to find out why.

99% is a common case
sometimes it helps:

  • check if there is enough disk space

  • uninstall sdkmanager, delete the sdkmanager folders including downloads, reinstall, try again
    Often the issue might be caused if the wrong value for target board is selected

Hi @WayneWWW and @Andrey1984

It will take some time before I can try your suggestions and get back to you regarding this topic.
I currently do not have access to the other NX and I will get it back early next week.

Regards,
Ritvik

Hi @Andrey1984,
There is enough space on the machine and i removed all sdkmanager folders and reinstalled sdkmanager and tried again. It got stuck again at 99%.

Hi @WayneWWW,
Here are the logs on the NX that i received towards the end over minicom:

[0289.393] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driver
[0289.401] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon' driver
[0289.403] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x46
[0289.413] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[0289.421] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[0289.425] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[0289.431] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x44
[0289.439] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[0289.447] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[0289.452] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[0289.461] I> fixed regulator driver initialized
[0289.474] I> CPU: Nvidia Carmel
[0289.474] I> CPU: MIDR: 0x4e0f0040, MPIDR: 0x80000000
[0289.474] I> chip revision : A02P
[0289.475] I> Boot-device: QSPI
[0289.477] I> Boot_device: QSPI_FLASH instance: 0
[0289.482] I> QSPI source rate = 204000 Khz
[0289.485] I> Requested rate for QSPI clock = 34000 Khz
[0289.490] I> BPMP-set rate for QSPI clk = 34000 Khz
[0289.495] I> QSPI Flash Size = 32 MB
[0289.503] I> Qspi initialized successfully
[0289.503] I> qspi flash-0 params source = boot args
[0289.507] I> Found sdcard
[0289.513] I> enabling 'vdd-sdmmc1-sw' regulator
[0289.521] I> regulator 'vdd-sdmmc1-sw' already enabled
[0289.526] E> Error in command_complete 18001 int_status
[0289.528] E> Error in command_complete c8001 int_status
[0289.531] E> Error in command_complete c8001 int_status
[0289.534] E> Sending CMD_SD_SEND_IF_COND failed
[0289.548] W> Error opening sdcard-0
[0289.549] I> -0 params source = 
[0289.549] E> Failed to initialize device 6-0
[0289.549] E> Top caller module: SDMMC, error module: SDMMC, reason: 0x17, aux_info: 0x01
[0289.557] I> TBoot-CPU Recovery hang

Just want to confirm.

Do both NX which you said have problem suffer the same error log during flash?

And are you using a NV devkit here?

Also, have you tried other SDcard?

  1. try headless sdkmanager
  2. what is the free space amount
  3. make sure you distinguish devkit nx versus production nx → set up correct target device

No, both NXs do not face this issue.
Only the first one from my original query faces this problem.
I was able to flash the other one using the SDKManager and it works fine now.

Yes, I am using the NV devkit here.
I have also tried with another SD Card.

  1. Can you please point me towards some documentation about how to use it in headless mode?
  2. The free space amount on the laptop is around 110GB and on the SD Card is 64GB
  3. Yes I have chosen the Devkit NX in the target device

try this

sdkmanager --cli install  --product Jetson --version 4.5.1 --targetos Linux --target P3668-0000 --flash all --targetimagefolder ~/Downloads --downloadfolder ~/Downloads_nx_devkit_v451

It is more like a hardware problem to me.

If there are two jetson NX, can we do some cross check here? For example, move the working SDcard to the board that does not work, run the same flash process on this setup and see if it can be flashed.

If it cannot, then I think it is not software side problem.

Even with the headless sdkmanager it ends up stuck at the same error message and stuck at 99%.
Using different SD Card does not work.

@WayneWWW @Andrey1984 if you have any more ideas i’d be glad to try them out in my off hours otherwise we can just let this go…sometimes things just break i guess!

Then please RMA this module and board. I guess it is not an issue that you can resolve from using software.