Jetson TX2 - does not boot and flashing fails

One of my Jetson TX2 developer board which was working fine so far does not boot at all now, seems like an early boot failure. The console does not show anything. The unit did boot up in Force Recovery mode and I initiated flashing using the flash.sh script (JetPack 3.2.1), it continued flashing for a while and then failed with these messages on the console -

[1465.233] I> BR-BCT relocated to 0xd7020000
[1465.237] I> Boot-device: eMMC
[1465.592] E> Error opening sdmmc-3
[1465.595] E> Failed to initialize boot device
[1465.600] C> ERROR: Highest Layer Module = 0xf, Lowest Layer Module = 0xf,
Aux Info = 0x2, Reason = 0x2
[1465.609] C> Hang

Searched for this error and found someone reported the exact same issue few months ago -
https://devtalk.nvidia.com/default/topic/1037473/jetson-tx2/cannot-flash-emmc-gt-error-opening-sdmmc-3/

Questions :

  1. Do we have any more information on what is causing this failure ?
  2. Is there a way to recover this unit at my end or do I have to return with RMA info ?
    Thanks

What was your exact command line for the flash?

Here it is :
sudo ./flash.sh jetson-tx2 mmcblk0p1

Thanks

This may be an actual failure. Any kind of flash attempt could have already changed eMMC content and invalidated current partitions, but I suspect partitions are still intact even if data has changed. What you might consider doing is attempting to clone the rootfs. I don’t say this for reasons of backup…any backup at this stage is likely pointless. However, reading eMMC in recovery mode might demonstrate part of the system functioning (parts which have fewer requirements than writing). Should the clone fail, then it is probably further evidence of actual hardware failure.

A clone can take a lot of space (it is an entire eMMC for the most part), so you might want to make sure your host has at least 35GB (or more) of free space wherever you clone (and that will be where the flash.sh script exists). The command “df -H /where/ever/you/clone” will provide that information for you. Details for cloning depend on release, but since you mention JetPack3.2.1 I am going to assume it is a recent release.

The Jetson would need to be put into recovery mode (and if any other operation was attempted, then put freshly back into recovery mode for each new command, e.g., you can’t clone twice without reentering recovery mode between attempts) and connected to the host via the micro-B USB cable. Verify “lsusb -d -0955:7c18” shows the recovery mode Jetson. Then, where your flash.sh exists:

sudo ./flash.sh -r -k APP -G my_backup.img jetson-tx2 mmcblk0p1

Some errors might exist which are not part of failed hardware, but I suspect any error from hardware failure would be more obvious than something like not finding a partition. If you can’t clone, then chances are RMA will be the only option.

Tried what you suggested and it failed.

Here are the console messages :

[0155.244] I> Welcome to MB1-recovery(prd-version : 14.01.171031-t186-M-00.00-6b33ecf5)
[0155.252] I> rst_source : 0x0
[0155.255] I> rst_level : 0x0
[0155.257] I> Read lock all AES keyslots
[0155.261] I> Read lock all RSA keyslots
[0155.265] I> Clear SE keyslots left open by BR
[0155.269] I> Boot-device: eMMC
[0155.272] I> Entering into 3p server..
[0155.276] I> Transport interface is USB
[0155.279] I> tegrabl_usbf_priv_init:2576 full init:: [0155.284] I>  HIGH SPEED
[0155.808] W> Get desc.Unsupported req: desc_index = 0
[0155.815] I> transport_usbf_priv_open USB configuration success
[0155.829] W> Get desc.Unsupported req: desc_index = 0
[0156.299] I> Populate eeprom info for module cvm
[0160.431] I> bit @ 0xd480000
[0160.434] I> BCT @ 0x0
[0160.436] I> Boot-device: eMMC
[0160.793] C> Error opening sdmmc-3
[0160.797] E> Failed to load bootrom bct
[0160.800] E> (0783c202): Failed to fetch load address

Here is the output from executing the command :

64_TX2/Linux_for_Tegra$ sudo ./flash.sh -r -k APP -G my_backup.img jetson-tx2 mmcblk0p1
###############################################################################
# L4T BSP Information:
# R28 (release), REVISION: 2.1, GCID: 11272647, BOARD: t186ref, EABI: aarch64, 
# DATE: Thu May 17 07:29:06 UTC 2018
###############################################################################
# Target Board Information:
# Name: jetson-tx2, Board Family: t186ref, SoC: Tegra 186, 
# OpMode: production, Boot Authentication: , 
###############################################################################
./tegraflash.py --chip 0x18 --applet "/home/test/64_TX2/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin" --cmd "dump eeprom boardinfo cvm.bin" --skipuid 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0044 ] Generating RCM messages
[   0.0056 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 --download rcm /home/test/64_TX2/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin 0 0
[   0.0065 ] RCM 0 is saved as rcm_0.rcm
[   0.0072 ] RCM 1 is saved as rcm_1.rcm
[   0.0072 ] List of rcm files are saved in rcm_list.xml
[   0.0072 ] 
[   0.0073 ] Signing RCM messages
[   0.0087 ] tegrasign_v2 --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0108 ] Assuming zero filled SBK key
[   0.0153 ] 
[   0.0154 ] Copying signature to RCM mesages
[   0.0165 ] tegrarcm_v2 --chip 0x18 --updatesig rcm_list_signed.xml
[   0.0179 ] 
[   0.0180 ] Boot Rom communication
[   0.0188 ] tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml --skipuid
[   0.0197 ] RCM version 0X180001
[   0.0208 ] Boot Rom communication completed
[   1.0283 ] 
[   1.0328 ] tegrarcm_v2 --isapplet
[   1.0362 ] Applet version 01.00.0000
[   1.0517 ] 
[   1.0554 ] Retrieving EEPROM data
[   1.0558 ] tegrarcm_v2 --oem platformdetails eeprom cvm /home/test/64_TX2/Linux_for_Tegra/bootloader/cvm.bin
[   1.0593 ] Applet version 01.00.0000
[   1.0841 ] Saved platform info in /home/test/64_TX2/Linux_for_Tegra/bootloader/cvm.bin
[   1.1625 ] 
Board ID(3310) version(B02) 
copying bctfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/P3310_A00_8GB_Samsung_8GB_lpddr4_204Mhz_A02_l4t.cfg)... done.
copying misc_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-misc-si-l4t.cfg)... done.
copying pinmux_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-pinmux-quill-p3310-1000-c03.cfg)... done.
copying pmic_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-pmic-quill-p3310-1000-c04.cfg)... done.
copying pmc_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-pad-quill-p3310-1000-c03.cfg)... done.
copying prod_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-prod-quill-p3310-1000-c03.cfg)... done.
copying scr_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/minimal_scr.cfg)... done.
copying scr_cold_boot_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/mobile_scr.cfg)... done.
copying bootrom_config(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-bootrom-quill-p3310-1000-c03.cfg)... done.
copying dev_params(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/emmc.cfg)... done.
Existing bootloader(/home/test/64_TX2/Linux_for_Tegra/bootloader/nvtboot_cpu.bin) reused.
	populating kernel to rootfs... done.
	populating initrd to rootfs... done.
	populating extlinux.conf.emmc to rootfs... done.
	populating /home/test/64_TX2/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb to rootfs... done.
done.
Making Boot image... done.
Existing sosfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin) reused.
copying tegraboot(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/nvtboot.bin)... done.
Existing mb2blfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/nvtboot_recovery.bin) reused.
Existing mtspreboot(/home/test/64_TX2/Linux_for_Tegra/bootloader/preboot_d15_prod_cr.bin) reused.
Existing mts(/home/test/64_TX2/Linux_for_Tegra/bootloader/mce_mts_d15_prod_cr.bin) reused.
Existing mb1file(/home/test/64_TX2/Linux_for_Tegra/bootloader/mb1_prod.bin) reused.
Existing bpffile(/home/test/64_TX2/Linux_for_Tegra/bootloader/bpmp.bin) reused.
copying bpfdtbfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2.dtb)... done.
Existing scefile(/home/test/64_TX2/Linux_for_Tegra/bootloader/camera-rtcpu-sce.bin) reused.
Existing spefile(/home/test/64_TX2/Linux_for_Tegra/bootloader/spe.bin) reused.
copying wb0boot(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/warmboot.bin)... done.
Existing tosfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/tos.img) reused.
Existing eksfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/eks.img) reused.
copying dtbfile(/home/test/64_TX2/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb)... done.
Skip generating system.img
Existing tbcfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/cboot.bin) reused.
copying tbcdtbfile(/home/test/64_TX2/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb)... done.
copying cfgfile(/home/test/64_TX2/Linux_for_Tegra/bootloader/t186ref/cfg/flash_l4t_t186.xml) to flash.xml... done.
Existing flasher(/home/test/64_TX2/Linux_for_Tegra/bootloader/nvtboot_recovery_cpu.bin) reused.
Existing flashapp(/home/test/64_TX2/Linux_for_Tegra/bootloader/tegraflash.py) reused.
*** Reading [APP] and storing to /home/test/64_TX2/Linux_for_Tegra/my_backup.img ***
./tegraflash.py --bl nvtboot_recovery_cpu.bin  --chip 0x18 --applet mb1_recovery_prod.bin --sdram_config P3310_A00_8GB_Samsung_8GB_lpddr4_204Mhz_A02_l4t.cfg --misc_config tegra186-mb1-bct-misc-si-l4t.cfg --pinmux_config tegra186-mb1-bct-pinmux-quill-p3310-1000-c03.cfg --pmic_config tegra186-mb1-bct-pmic-quill-p3310-1000-c04.cfg --pmc_config tegra186-mb1-bct-pad-quill-p3310-1000-c03.cfg --prod_config tegra186-mb1-bct-prod-quill-p3310-1000-c03.cfg --scr_config minimal_scr.cfg --scr_cold_boot_config mobile_scr.cfg --br_cmd_config tegra186-mb1-bct-bootrom-quill-p3310-1000-c03.cfg --dev_params emmc.cfg  --cfg  flash.xml --bins "mb2_bootloader nvtboot_recovery.bin; mts_preboot preboot_d15_prod_cr.bin; mts_bootpack mce_mts_d15_prod_cr.bin; bpmp_fw bpmp.bin; bpmp_fw_dtb tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2.dtb; tlk tos.img; eks eks.img; bootloader_dtb tegra186-quill-p3310-1000-c03-00-base.dtb"  --cmd "read APP /home/test/64_TX2/Linux_for_Tegra/my_backup.img"  
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0102 ] Generating RCM messages
[   0.0112 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 --download rcm mb1_recovery_prod.bin 0 0
[   0.0120 ] RCM 0 is saved as rcm_0.rcm
[   0.0123 ] RCM 1 is saved as rcm_1.rcm
[   0.0125 ] List of rcm files are saved in rcm_list.xml
[   0.0125 ] 
[   0.0126 ] Signing RCM messages
[   0.0133 ] tegrasign_v2 --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0140 ] Assuming zero filled SBK key
[   0.0177 ] 
[   0.0177 ] Copying signature to RCM mesages
[   0.0184 ] tegrarcm_v2 --chip 0x18 --updatesig rcm_list_signed.xml
[   0.0196 ] 
[   0.0196 ] Boot Rom communication
[   0.0204 ] tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml
[   0.0212 ] BootRom is not running
[   1.0200 ] 
[   1.0235 ] tegrarcm_v2 --isapplet
[   1.0264 ] Applet version 01.00.0000
[   1.0384 ] 
[   1.0418 ] tegrasign_v2 --key None --getmode mode.txt
[   1.0446 ] Assuming zero filled SBK key
[   1.0452 ] 
[   1.0453 ] Parsing partition layout
[   1.0482 ] tegraparser_v2 --pt flash.xml.tmp
[   1.0525 ] 
[   1.0526 ] Creating list of images to be signed
[   1.0553 ] tegrahost_v2 --chip 0x18 --partitionlayout flash.xml.bin --list images_list.xml zerosbk
[   1.1174 ] 
[   1.1174 ] Generating signatures
[   1.1184 ] tegrasign_v2 --key None --list images_list.xml --pubkeyhash pub_key.key
[   1.1193 ] Assuming zero filled SBK key
[   1.3514 ] 
[   1.3514 ] Reading BCT from device for further operations
[   1.3514 ] Generating blob
[   1.3523 ] tegrahost_v2 --align blob_nvtboot_recovery_cpu.bin
[   1.3530 ] 
[   1.3537 ] tegrahost_v2 --appendsigheader blob_nvtboot_recovery_cpu.bin zerosbk
[   1.3548 ] 
[   1.3557 ] tegrasign_v2 --key None --list blob_nvtboot_recovery_cpu_sigheader.bin_list.xml
[   1.3563 ] Assuming zero filled SBK key
[   1.3607 ] 
[   1.3615 ] tegrahost_v2 --updatesigheader blob_nvtboot_recovery_cpu_sigheader.bin.encrypt blob_nvtboot_recovery_cpu_sigheader.bin.hash zerosbk
[   1.3627 ] 
[   1.3636 ] tegrahost_v2 --align blob_nvtboot_recovery.bin
[   1.3642 ] 
[   1.3649 ] tegrahost_v2 --appendsigheader blob_nvtboot_recovery.bin zerosbk
[   1.3658 ] 
[   1.3667 ] tegrasign_v2 --key None --list blob_nvtboot_recovery_sigheader.bin_list.xml
[   1.3673 ] Assuming zero filled SBK key
[   1.3693 ] 
[   1.3701 ] tegrahost_v2 --updatesigheader blob_nvtboot_recovery_sigheader.bin.encrypt blob_nvtboot_recovery_sigheader.bin.hash zerosbk
[   1.3710 ] 
[   1.3717 ] tegrahost_v2 --align blob_preboot_d15_prod_cr.bin
[   1.3724 ] 
[   1.3730 ] tegrahost_v2 --appendsigheader blob_preboot_d15_prod_cr.bin zerosbk
[   1.3739 ] 
[   1.3747 ] tegrasign_v2 --key None --list blob_preboot_d15_prod_cr_sigheader.bin_list.xml
[   1.3754 ] Assuming zero filled SBK key
[   1.3769 ] 
[   1.3778 ] tegrahost_v2 --updatesigheader blob_preboot_d15_prod_cr_sigheader.bin.encrypt blob_preboot_d15_prod_cr_sigheader.bin.hash zerosbk
[   1.3787 ] 
[   1.3795 ] tegrahost_v2 --align blob_mce_mts_d15_prod_cr.bin
[   1.3802 ] 
[   1.3809 ] tegrahost_v2 --appendsigheader blob_mce_mts_d15_prod_cr.bin zerosbk
[   1.3869 ] 
[   1.3881 ] tegrasign_v2 --key None --list blob_mce_mts_d15_prod_cr_sigheader.bin_list.xml
[   1.3888 ] Assuming zero filled SBK key
[   1.4394 ] 
[   1.4405 ] tegrahost_v2 --updatesigheader blob_mce_mts_d15_prod_cr_sigheader.bin.encrypt blob_mce_mts_d15_prod_cr_sigheader.bin.hash zerosbk
[   1.4463 ] 
[   1.4473 ] tegrahost_v2 --align blob_bpmp.bin
[   1.4481 ] 
[   1.4489 ] tegrahost_v2 --appendsigheader blob_bpmp.bin zerosbk
[   1.4510 ] 
[   1.4520 ] tegrasign_v2 --key None --list blob_bpmp_sigheader.bin_list.xml
[   1.4527 ] Assuming zero filled SBK key
[   1.4657 ] 
[   1.4668 ] tegrahost_v2 --updatesigheader blob_bpmp_sigheader.bin.encrypt blob_bpmp_sigheader.bin.hash zerosbk
[   1.4689 ] 
[   1.4697 ] tegrahost_v2 --align blob_tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2.dtb
[   1.4705 ] 
[   1.4712 ] tegrahost_v2 --appendsigheader blob_tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2.dtb zerosbk
[   1.4732 ] 
[   1.4741 ] tegrasign_v2 --key None --list blob_tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2_sigheader.dtb_list.xml
[   1.4748 ] Assuming zero filled SBK key
[   1.4857 ] 
[   1.4866 ] tegrahost_v2 --updatesigheader blob_tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2_sigheader.dtb.encrypt blob_tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2_sigheader.dtb.hash zerosbk
[   1.4883 ] 
[   1.4891 ] tegrahost_v2 --align blob_tos.img
[   1.4898 ] 
[   1.4905 ] tegrahost_v2 --appendsigheader blob_tos.img zerosbk
[   1.4915 ] 
[   1.4924 ] tegrasign_v2 --key None --list blob_tos_sigheader.img_list.xml
[   1.4930 ] Assuming zero filled SBK key
[   1.4947 ] 
[   1.4956 ] tegrahost_v2 --updatesigheader blob_tos_sigheader.img.encrypt blob_tos_sigheader.img.hash zerosbk
[   1.4964 ] 
[   1.4972 ] tegrahost_v2 --align blob_eks.img
[   1.4980 ] 
[   1.4987 ] tegrahost_v2 --appendsigheader blob_eks.img zerosbk
[   1.4995 ] 
[   1.5005 ] tegrasign_v2 --key None --list blob_eks_sigheader.img_list.xml
[   1.5011 ] Assuming zero filled SBK key
[   1.5014 ] 
[   1.5023 ] tegrahost_v2 --updatesigheader blob_eks_sigheader.img.encrypt blob_eks_sigheader.img.hash zerosbk
[   1.5031 ] 
[   1.5039 ] tegrahost_v2 --align blob_tegra186-quill-p3310-1000-c03-00-base.dtb
[   1.5046 ] 
[   1.5053 ] tegrahost_v2 --appendsigheader blob_tegra186-quill-p3310-1000-c03-00-base.dtb zerosbk
[   1.5068 ] 
[   1.5078 ] tegrasign_v2 --key None --list blob_tegra186-quill-p3310-1000-c03-00-base_sigheader.dtb_list.xml
[   1.5086 ] Assuming zero filled SBK key
[   1.5159 ] 
[   1.5169 ] tegrahost_v2 --updatesigheader blob_tegra186-quill-p3310-1000-c03-00-base_sigheader.dtb.encrypt blob_tegra186-quill-p3310-1000-c03-00-base_sigheader.dtb.hash zerosbk
[   1.5184 ] 
[   1.5195 ] tegrahost_v2 --generateblob blob.xml blob.bin
[   1.5203 ] number of images in blob are 9
[   1.5204 ] blobsize is 3749384
[   1.5205 ] Added binary blob_nvtboot_recovery_cpu_sigheader.bin.encrypt of size 190048
[   1.5220 ] Added binary blob_nvtboot_recovery_sigheader.bin.encrypt of size 79600
[   1.5222 ] Added binary blob_preboot_d15_prod_cr_sigheader.bin.encrypt of size 63024
[   1.5225 ] Added binary blob_mce_mts_d15_prod_cr_sigheader.bin.encrypt of size 2076832
[   1.5232 ] Added binary blob_bpmp_sigheader.bin.encrypt of size 529440
[   1.5237 ] Added binary blob_tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2_sigheader.dtb.encrypt of size 466160
[   1.5241 ] Added binary blob_tos_sigheader.img.encrypt of size 62976
[   1.5243 ] Added binary blob_eks_sigheader.img.encrypt of size 1488
[   1.5246 ] Added binary blob_tegra186-quill-p3310-1000-c03-00-base_sigheader.dtb.encrypt of size 279664
[   1.5315 ] 
[   1.5315 ] Sending bootloader and pre-requisite binaries
[   1.5324 ] tegrarcm_v2 --download blob blob.bin
[   1.5331 ] Applet version 01.00.000
[   1.5468 ] Sending blob
[   1.5470 ] 000000000783c202: Failed to fetch load address
[   1.9226 ] 
Error: Return value 2
Command tegrarcm_v2 --download blob blob.bin
*** The [APP] has been read successfully. ***
	Converting RAW image to Sparse image... 
mv: cannot stat '/home/test/jetpack_download/64_TX2/Linux_for_Tegra/my_backup.img': No such file or directory
open input file /home/test/jetpack_download_hardening/64_TX2/Linux_for_Tegra/my_backup.img.raw failed.

It sounds like even basic read of eMMC is failing. It is probably time to RMA. For RMA information you can search for “RMA” near the top of this:
https://devtalk.nvidia.com/default/topic/793798/embedded-systems/some-jetson-web-links/

Thank You. Will start with RMA procedure.