TX2i now fails to flash (or boot) Dead emmc?

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

We recently had a TX2i that was not booting properly (no information being posted to the system console). We hooked up to it and attempted to flash it, which fails, it looks like maybe the emmc is toast.

Console During flash attempt:

[0009.660] I> rst_source : 0x0
[0009.663] I> rst_level : 0x0
[0009.665] I> Read lock all AES keyslots
[0009.669] I> Read lock all RSA keyslots
[0009.673] I> Clear SE keyslots left open by BR
[0009.677] I> Boot-device: eMMC
[0009.680] I> Entering into 3p server..
[0009.684] I> Transport interface is USB
[0009.687] I> tegrabl_usbf_priv_init:2576 full init:: [0009.692] I>  HIGH SPEED
[0010.041] W> Get desc.Unsupported req: desc_index = 0
[0010.048] I> transport_usbf_priv_open USB configuration success
[0010.700] I> Populate eeprom info for module cvm
[0087.734] I> bit @ 0xd480000
[0087.737] I> Downloading bct_bootrom at 0x000000004004e800
[0087.744] I> downloaded image bct_bootrom successfully
[0087.750] I> Downloading bct_mb1 at 0x0000000040000000
[0087.761] I> MB1-BCT relocated to 0xd48bc00 from 0x40000000
[0087.766] I> Supported mb1-bct versions 9-14
[0087.770] I> Version of mb1 bct binary 14
[0087.774] W> MB1 BCT does not have controller prod config data
[0087.780] I> Bootrom patch version : 31 (correctly patched)
[0087.785] I> ATE fuse revision : 0x518a0
[0087.789] I> chip revision : A02P
[0087.792] I> disabled_core_mask: 0xffffff0c
[0087.796] I> Enabled Cores: 0xf3
[0087.799] I> TPC disable fuse status: 0
[0087.803] I> Booting Parker Sku
[0087.806] I> Ram repair fuse : 0x0
[0087.809] I> Mem length:1c00
[0087.812] I> tegrabl_ast_init_mapper: AST-cache dump
[0087.817] I> allocated(CO:30) base:0x238000000 size:0x8000000 align: 0x8000000
[0087.824] I> allocated(CO:23) base:0xf0100000 size:0x10000 align: 0x10000
[0087.831] I> allocated(CO:31) base:0x0 size:0x0 align: 0x0
[0087.836] I> allocated(CO:1) base:0x237f00000 size:0x100000 align: 0x100000
[0087.843] I> allocated(CO:2) base:0x237e00000 size:0x100000 align: 0x100000
[0087.850] I> allocated(CO:3) base:0x237d00000 size:0x100000 align: 0x100000
[0087.857] I> allocated(CO:4) base:0x237c00000 size:0x100000 align: 0x100000
[0087.864] I> allocated(CO:5) base:0x237b00000 size:0x100000 align: 0x100000
[0087.871] I> allocated(CO:27) base:0x0 size:0x0 align: 0x0
[0087.876] I> allocated(CO:6) base:0x237800000 size:0x200000 align: 0x200000
[0087.883] I> allocated(CO:7) base:0x237400000 size:0x400000 align: 0x400000
[0087.890] I> allocated(CO:8) base:0x237a00000 size:0x100000 align: 0x100000
[0087.897] I> allocated(CO:9) base:0x237300000 size:0x100000 align: 0x100000
[0087.904] I> allocated(CO:10) base:0x236800000 size:0x800000 align: 0x800000
[0087.911] I> allocated(CO:11) base:0x30000000 size:0x40000 align: 0x0
[0087.917] I> allocated(CO:12) base:0xf0000000 size:0x100000 align: 0x100000
[0087.924] I> allocated(CO:13) base:0x30040000 size:0x1000 align: 0x0
[0087.930] I> allocated(CO:14) base:0x30048000 size:0x1000 align: 0x0
[0087.936] I> allocated(CO:15) base:0x30049000 size:0x1000 align: 0x0
[0087.943] I> allocated(CO:16) base:0x3004a000 size:0x1000 align: 0x0
[0087.949] I> allocated(CO:17) base:0x3004b000 size:0x1000 align: 0x0
[0087.955] I> allocated(CO:19) base:0x3004d000 size:0x1000 align: 0x0
[0087.961] I> allocated(CO:18) base:0x3004c000 size:0x1000 align: 0x0
[0087.968] I> allocated(CO:20) base:0x3004e000 size:0x1000 align: 0x0
[0087.974] I> allocated(CO:21) base:0x3004f000 size:0x1000 align: 0x0
[0087.980] I> allocated(CO:22) base:0x0 size:0x0 align: 0x0
[0087.986] I> allocated(CO:32) base:0x236000000 size:0x600000 align: 0x400000
[0087.992] I> allocated(CO:24) base:0x0 size:0x0 align: 0x0
[0087.998] I> allocated(CO:25) base:0x0 size:0x0 align: 0x0
[0088.003] I> allocated(CO:26) base:0x0 size:0x0 align: 0x0
[0088.009] I> allocated(CO:39) base:0x96000000 size:0x2400000 align: 0x2000000
[0088.016] I> allocated(CO:38) base:0x84000000 size:0x400000 align: 0x400000
[0088.022] I> allocated(CO:37) base:0x2372e0000 size:0x20000 align: 0x20000
[0088.029] I> allocated(CO:40) base:0x85000000 size:0x1200000 align: 0x1000000
[0088.036] I> allocated(CO:41) base:0x237000000 size:0x280000 align: 0x200000
[0088.043] I> allocated(CO:28) base:0x84400000 size:0x400000 align: 0x400000
[0088.051] W> MB1 PMIC CFG is version 1.2
[0088.055] W> Rail ID 7 is not found on MB1 BCT
[0088.059] W> MB1 PMIC CFG is version 1.2
[0088.065] W> MB1 PMIC CFG is version 1.2
[0088.072] W> MB1 PMIC CFG is version 1.2
[0088.076] I> sdram slot : 0
[0088.079] I> Initializing SDRAM
[0088.085] I> SDRAM init scrub in progress...
[0097.300] I> SDRAM scrub DONE
[0097.303] I> SDRAM scrub successful
[0097.307] I> mss encrypt status check successful
[0097.311] I> downloaded image bct_mb1 successfully
[0097.639] I> Downloading blob at 0x0000000084800000
[0098.032] I> downloaded image blob successfully
[0098.056] I> exiting nv3p server
[0098.059] I> MB1-recovery(prd-version : 14.01.171031-t186-M-00.00-6b33ecf5)
[0098.067] I> disabled_core_mask: 0xffffff0c
[0098.071] I> Enabled Cores: 0xf3
[0098.074] I> TPC disable fuse status: 0
[0098.077] I> Booting Parker Sku
[0098.080] I> Setup temp. GSC#29 (base:0x30000000, size:0x00010000)
[0098.087] I> Loading preboot
[0098.091] I> Binary(1) of size 62224 is loaded @ 0x40000000
[0098.097] I> Restrict GSC#29 access to DPMU only
[0098.101] I> Updated LSR_DVCOMP_PARAMS: 0xc07c0f85c
[0098.106] I> Updated LSR_DVCOMP_PARAMS: 0xc07c0f85c
[0098.111] I> WP0 triggered
[0098.113] I> Waypoint-0 ACK received :)
[0098.117] I> WP0.5 triggered
[0098.121] I> Initialized SCRs
[0098.123] W> MB1 PMIC CFG is version 1.2
[0098.131] W> MB1 PMIC CFG is version 1.2
[0098.201] I> Waypoint-0.5 ACK received :)
[0098.205] I> Setup temp. GSC#28 (base:0x84400000, size:0x00400000)
[0098.211] I> Loading MTS-DMCE
[0098[0098.231] I> disabled_core_mask: 0xffffff0c
[0098.235] I> Loading MTS-PROPER
[0098.238] I> Binary(3) of size 1810512 is loaded @ 0x0
[0098.243] I> Restrict GSC#28 access to DPMU only
[0098.248] I> disabled_core_mask: 0xffffff0c
[0098.252] I> WP1 triggered
[0098.254] I> Loading MB2
[0098.258] I> Binary(8) of size 79200 is loaded @ 0x84000000
[0098.263] W> MB1 PMIC CFG is version 1.2
[0098.268] I> Configure DMCE carveout GSC#13
[0098.272] I> CPU reset vector: 0x96000000
[0098.276] I> MB2-params @ 0xd7000000
[0098.280] I> Waypoint-1 ACK received :)
[0098.284] I> Configure SC7 carveout GSC#23
[0098.288] I> MB1 (recovery) done 

[0098.294] I> Welcome to MB2(TBoot-BPMP) Recovery(version: 01.00.160913-t186-M-00.00-mobile-cfa59593)
[0098.303] I> Default Heap @ [0xd486400 - 0xd488400]
[0098.308] I> DMA Heap @ [0x85200000 - 0x86200000]
[0098.313] I> bit @ 0xd480000
[0098.315] I> BR-BCT relocated to 0xd7020000
[0098.320] I> Boot-device: eMMC
[0098.360] E> Error opening sdmmc-3
[0098.363] E> Failed to initialize boot device
[0098.368] C> ERROR: Highest Layer Module = 0xf, Lowest Layer Module = 0xf,
Aux Info = 0x2, Reason = 0x2
[0098.377] C> Hang
.226] I> Binary(2) of size 265472 is loaded @ 0x84400000

The flash console screen looks normal, but it hangs at this point:

[  11.5699 ] Sending bootloader and pre-requisite binaries
[  11.5706 ] tegrarcm_v2 --download blob blob.bin
[  11.5712 ] Applet version 01.00.0000
[  11.5833 ] Sending blob
[  11.5834 ] [................................................] 100%
[  11.9835 ] 
[  11.9855 ] tegrarcm_v2 --boot recovery
[  11.9874 ] Applet version 01.00.0000
[  12.0017 ] 
[  12.0035 ] tegrarcm_v2 --isapplet

Any suggestions?

#2

hello SrJsignal,

couple of suggestion for your testing,

  1. may I have more details, could you please share your commands that you’re used for flashing.
  2. since you’re still able to enter forced-recovery, are you able to flash last-known-good image to the target.
#3

  1. ./flash.sh jetson-tx2i mmcblk0p1

  2. This is a known-good image.

The system stopped responding (booting) one day this week. Nothing was coming out on the console, so as a last ditch effort we attempted to flash a known good image. I have the exact image up and running on a different tx2i.

#4

Hi, are you using custom carrier board? If so, which version design did you follow, B04 or C02? Please check OEM DG for detail design difference between them.

#5

This occurs on either our custom board or the devkit board. Also, I need to reiterate, this system was working fine for months, another TX2i is currently working on this exact carrier board, with the same firmware / OS load.

The power supply sequencing / discharge is not an exact match for either B04 or C02, I do not see anything else in the design guides showing much difference. Our design is controlled by a microcontroller, and in general does not use the power button functionality. The pin is held high.

My goal is really to just understand what failed so we can do a failure analysis and risk analysis for using this product in our designs. If the eMMC is a weak point, that is not a good thing at all. If on the other hand this is an “infant mortality” issue, then we need to make sure our pre-ship testing runs the eMMC long enough to show early failures.

#6

If your other TX2i can work on same carrier board, then the previous TX2i module might already be broken. You might need RMA, there might be a report on this to show the root cause.