AGX Orin + CTI Forge Flashing - Parsing Board Information Failed

We are using CTI’s Forge Carrier board + AGX Orin 32GB and we are seeing the “Parsing board information failed” error after running the flash script. Here are the log messages from both the host and device side:

Device side:

labuser@lab-desktop-19:~$ sudo picocom -b 115200 /dev/ttyUSB0
picocom v3.1

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
hangup is      : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        : 
omap is        : 
emap is        : crcrlf,delbs,
logfile is     : none
initstring     : none
exit_after is  : not set
exit is        : no

Type [C-a] [C-h] to see available commands
Terminal ready
��
[0109.806] I> MB1 (version: 0.28.0.0-t234-54845784-ec016368)
[0109.812] I> t234-A01-0-Silicon (0x12347) Prod
[0109.816] I> Boot-mode : BPMP Diagnostic
[0109.820] I> Emulation: 
[0109.822] I> Entry timestamp: 0x00000000
[0109.826] I> last_boot_error: 0x0
[0109.829] I> BR-BCT: preprod_dev_sign: 0
[0109.833] I> rst_source: 0xb, rst_level: 0x1
[0109.837] I> Task: Initialize MB2 params (0x5000d78d)
[0109.842] I> MB2-params @ 0x40060000
[0109.846] I> Task: Crypto init (0x5001cb8d)
[0109.850] I> Task: Secure debug controls (0x5000cd21)
[0109.855] I> Task: strap war set (0x5000c70d)
[0109.859] I> strap value(0x4000401) set to 0x4000401
[0109.864] I> Task: Program NV master stream id (0x5000ccd5)
[0109.869] I> Task: Verify boot mode (0xd482021)
[0109.875] I> Task: Alias fuses (0x500109cd)
[0109.880] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0109.887] I> Task: Print SKU type (0x5000fee1)
[0109.891] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x000001c0
[0109.897] I> FUSE_OPT_GPC_DISABLE = 0x00000000
[0109.901] I> FUSE_OPT_TPC_DISABLE = 0x00000008
[0109.905] I> FUSE_OPT_DLA_DISABLE = 0x00000000
[0109.909] I> FUSE_OPT_EMC_DISABLE = 0x00000000
[0109.914] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0109.918] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0109.922] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0109.926] I> FUSE_SKU_INFO_0 = 0xd2
[0109.929] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS 
[0109.934] I> FUSE_PACKAGE_INFO_0 = 0x2
[0109.937] I> SKU: Prod
[0109.939] I> Task: Boost clocks (0x5001462d)
[0109.944] I> Initializing PLLC2 for AXI_CBB.
[0109.948] I> AXI_CBB : src = 35, divisor = 0
[0109.952] I> Task: Voltage monitor (0x50014639)
[0109.956] I> VMON: Vmon re-calibration and fine tuning done
[0109.962] I> Task: UPHY init (0x5000dced)
[0109.966] W> UPHY: UPHY lane info table is empty in MB1 BCT.
[0109.971] I> Task: Boot device init (0x50000b39)
[0109.976] I> Boot_device: RCM
[0109.979] I> USB configuration success
[0109.982] I> Task: TSC init (0x5001fa6d)
[0109.986] I> Task: Enable WDT 5th expiry (0x50020699)
[0109.991] I> Task: I2C register (0x50000ad9)
[0109.995] I> Task: Reset FSI (0x50014635)
[0109.999] I> Task: Enable clock-mon (0x5001fa5d)
[0110.015] I> FMON: Fmon re-programming done
[0110.019] I> Task: MB1 fixed firewalls (0x5001f0d9)
[0110.028] I> Task: Load MB2/Applet/FSKP (0x5000d691)
[0110.032] I> Loading MB2 Applet
[0110.035] I> Active chain: 0
[0110.038] I> Slot: 0
[0110.040] I> Binary[21] block-0 (partition size: 0x50000)
[0110.045] I>  get_binary_info: Binary name: MB2-Applet
[0110.050] I> Size of crypto header is 8192
[0110.054] I> BCH load address is : 0x4004e000
[0110.058] I> Size of crypto header is 8192
[0110.063] I> BCH of MB2-Applet read from storage
[0110.067] I> BCH address is : 0x4004e000
[0110.071] I> MB2-Applet header integrity check is success
[0110.076] I> Binary magic in BCH component 0 is MB2A
[0110.081] I> component binary type is 21
[0110.085] I> Size of crypto header is 8192
[0110.096] I> MB2-Applet binary is read from storage
[0110.102] I> MB2-Applet binary integrity check is success
[0110.107] I> Binary MB2-Applet loaded successfully at 0x40000000 (0x41b40)
[0110.114] I> Task: Map CCPLEX SHARED carveout (0x5000d835)
[0110.119] I> Task: Prepare MB2 params (0x5000d8f1)
[0110.124] I> BR-BCT Boot Chain Fields
[0110.128] I> 	 u32_non_gpio_select_boot_chain  : 0
[0110.132] I> 	 u32_num_boot_chains             : 0
[0110.137] I> 	 bf_bl_gpio_select_boot_chain_1b : 0
[0110.141] I> Task: Misc NV security settings (0x5000d0f9)
[0110.147] I> NVDEC sticky bits programming done
[0110.151] I> Successfully powergated NVDEC
[0110.155] I> Task: Disable/Reload WDT (0x500206e1)
[0110.160] I> Programmed SLCG global override := 0x0
[0110.165] I> MB1: MSS reconfig completed
I> Applet (version: 0.0.0.0-t234-54845784-80607037)
I> t234-A01-0-Silicon (0x12347)
I> Emulation: 
I> Entry timestamp: 0xffffffff
I> Task: Pinmux init (0x4000062d)
I> Task: Boot device init (0x400012e5)
I> Applet do storage init
I> Boot_device: QSPI_FLASH instance: 0
I> Qspi clock source : clk_m
I> QSPI-0l initialized successfully
I> sdmmc DDR50 mode
I> sdmmc bdev is already initialized
E> Unknown device 7
E> Unknown device 8
I> Task: Partition Manager Init (0x40002d25)
I> Found 57 partitions in QSPI_FLASH (instance 0)
W> Cannot find any partition table for 00000003
 > PARTITION_MANAGER: Failed to publish partition.
I> Found 14 partitions in SDMMC_USER (instance 3)
I> Task: I2C register (0x400006a5)
I> Task: Register exit handlers (0x400005f5)
I> Task: Enter 3p server (0x400018e9)
I> USB configuration success
I> Populate chip info
I> RAM_CODE 0x4000401
I> Populate eeprom info
I> Populate eeprom info for module cvm
I> dump bct
I> Rebooting : reboot-recovery

Host side:

# L4T BSP Information:
# R35 , REVISION: 2.1
# User release: 0.0
###############################################################################
# Target Board Information:
# Name: cti-orin-agx-agx201-09, Board Family: t186ref, SoC: Tegra 234, 
# OpMode: production, Boot Authentication: NS, 
# Disk encryption: disabled ,
###############################################################################
copying emc_fuse_dev_params(/home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-br-bct-diag-boot.dts)... done.
copying device_config(/home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-mb1-bct-device-p3701-0000.dts)... done.
copying misc_config(/home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-mb1-bct-misc-p3701-0000.dts)... done.
./tegraflash.py --chip 0x23 --applet "/home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/mb1_t234_prod.bin" --skipuid --cfg readinfo_t234_min_prod.xml --dev_params tegra234-br-bct-diag-boot.dts --device_config tegra234-mb1-bct-device-p3701-0000.dts --misc_config tegra234-mb1-bct-misc-p3701-0000.dts --bins "mb2_applet applet_t234.bin" --cmd "dump eeprom cvm cvm.bin; dump custinfo custinfo_out.bin; reboot recovery" 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0121 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.0125 ] File rcm_state open failed
[   0.0127 ] ERROR: failed to read rcm_state
[   0.0127 ] 
[   0.0152 ] tegrasign_v3.py --key None --getmode mode.txt
[   0.0154 ] Assuming zero filled SBK key
[   0.0129 ] Pre-processing config: tegra234-mb1-bct-device-p3701-0000.dts
[   0.0195 ] Pre-processing config: tegra234-mb1-bct-misc-p3701-0000.dts
[   0.0282 ] Parsing partition layout
[   0.0286 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0298 ] Kernel DTB used: None
[   0.0298 ] WARNING: dce base dtb is not provided

[   0.0298 ] Parsing partition layout
[   0.0301 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0307 ] Creating list of images to be signed
[   0.0310 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --list images_list.xml zerosbk
[   0.0313 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0324 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0327 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0370 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0374 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0413 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0416 ] adding BCH for mb2_t234_aligned.bin
[   0.0440 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0548 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0550 ] adding BCH for mb2_t234_aligned.bin
[   0.0677 ] Filling MB1 storage info
[   0.0677 ] Parsing dev params for multi chains
[   0.0733 ] Generating br-bct
[   0.0736 ] Updating dev and MSS params in BR BCT
[   0.0736 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.0742 ] Updating bl info
[   0.0745 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin
[   0.0748 ] WARNING: boot chain is not completed. set to 0
[   0.0755 ] Generating signatures
[   0.0781 ] tegrasign_v3.py --key None --list images_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0782 ] Assuming zero filled SBK key
[   0.0860 ] Warning: pub_key.key is not found
[   0.0836 ] Parsing dev params for multi chains
[   0.0836 ] Generating br-bct
[   0.0839 ] Updating dev and MSS params in BR BCT
[   0.0839 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.0844 ] Updating bl info
[   0.0847 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.0849 ] WARNING: boot chain is not completed. set to 0
[   0.0857 ] Get Signed section of bct
[   0.0860 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --listbct bct_list.xml
[   0.0864 ] Signing BCT
[   0.0890 ] tegrasign_v3.py --key None --list bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0891 ] Assuming zero filled SBK key
[   0.0904 ] Sha saved in br_bct_BR.sha
[   0.0906 ] Warning: pub_key.key is not found
[   0.0880 ] Updating BCT with signature
[   0.0883 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesig bct_list_signed.xml
[   0.0885 ] Offset :4608 Len :3584
[   0.0889 ] Generating SHA2 Hash
[   0.0915 ] tegrasign_v3.py --key None --list bct_list.xml --sha sha512
[   0.0915 ] Assuming zero filled SBK key
[   0.0915 ] Assuming zero filled SBK key
[   0.0930 ] Sha saved in br_bct_BR.sha
[   0.0906 ] Updating BCT with SHA2 Hash
[   0.0909 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesha bct_list_signed.xml
[   0.0911 ] Offset :4608 Len :3584
[   0.0914 ] Offset :68 Len :8124
[   0.0916 ] Generating coldboot mb1-bct
[   0.0919 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct.cfg --misc tegra234-mb1-bct-misc-p3701-0000_cpp.dtb --device tegra234-mb1-bct-device-p3701-0000_cpp.dtb
[   0.0922 ] MB1-BCT version: 0.9
[   0.0924 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[   0.0927 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[   0.0929 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[   0.0931 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[   0.0933 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[   0.0935 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported

[   0.0940 ] Parsing config file :tegra234-mb1-bct-device-p3701-0000_cpp.dtb 
[   0.0941 ] Added Platform Config 9 data with size :- 100
[   0.0941 ] 
[   0.0941 ] Updating mb1-bct with firmware information
[   0.0944 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct_MB1.bct --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.0954 ] tegrahost_v2 --chip 0x23 0 --align mb1_cold_boot_bct_MB1_aligned.bct
[   0.0961 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_cold_boot_bct_MB1_aligned.bct zerosbk
[   0.0963 ] adding BCH for mb1_cold_boot_bct_MB1_aligned.bct
[   0.0994 ] tegrasign_v3.py --key None --list mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0995 ] Assuming zero filled SBK key
[   0.1005 ] Warning: pub_key.key is not found
[   0.0983 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_cold_boot_bct_MB1_aligned_sigheader.bct.encrypt mb1_cold_boot_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.0990 ] Generating recovery mb1-bct
[   0.0993 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct.cfg --misc tegra234-mb1-bct-misc-p3701-0000_cpp.dtb --device tegra234-mb1-bct-device-p3701-0000_cpp.dtb
[   0.0996 ] MB1-BCT version: 0.9
[   0.0998 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[   0.1000 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[   0.1002 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[   0.1004 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[   0.1006 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[   0.1008 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[   0.1009 ] 
[   0.1009 ] Parsing config file :tegra234-mb1-bct-device-p3701-0000_cpp.dtb 
[   0.1009 ] Added Platform Config 9 data with size :- 100
[   0.1009 ] 
[   0.1009 ] Updating mb1-bct with firmware information
[   0.1011 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct_MB1.bct --recov --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.1021 ] tegrahost_v2 --chip 0x23 0 --align mb1_bct_MB1_aligned.bct
[   0.1027 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_bct_MB1_aligned.bct zerosbk
[   0.1029 ] adding BCH for mb1_bct_MB1_aligned.bct
[   0.1060 ] tegrasign_v3.py --key None --list mb1_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1061 ] Assuming zero filled SBK key
[   0.1069 ] Warning: pub_key.key is not found
[   0.1047 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_bct_MB1_aligned_sigheader.bct.encrypt mb1_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.1054 ] Error: Skip generating mem_bct because sdram_config is not defined
[   0.1054 ] Error: Skip generating mem_bct because sdram_config is not defined
[   0.1054 ] Copying signatures
[   0.1056 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.1100 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.1101 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.1101 ] Boot Rom communication
[   0.1104 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[   0.1107 ] BR_CID: 0x80012344705DD3DF0800000017038140
[   0.1390 ] Sending bct_br
[   0.1868 ] Sending mb1
[   0.1878 ] Sending psc_bl1
[   0.1985 ] Sending bct_mb1
[   0.2045 ] Boot Rom communication completed
[   0.2064 ] tegrahost_v2 --chip 0x23 0 --align applet_t234_aligned.bin
[   0.2085 ] tegrahost_v2 --chip 0x23 0 --magicid MB2A --appendsigheader applet_t234_aligned.bin zerosbk
[   0.2093 ] adding BCH for applet_t234_aligned.bin
[   0.2338 ] tegrasign_v3.py --key None --list applet_t234_aligned_sigheader.bin_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.2342 ] Assuming zero filled SBK key
[   0.2386 ] Warning: pub_key.key is not found
[   0.2371 ] tegrahost_v2 --chip 0x23 0 --updatesigheader applet_t234_aligned_sigheader.bin.encrypt applet_t234_aligned_sigheader.bin.hash zerosbk
[   0.2404 ] Sending mb2_applet...

[   0.2412 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download applet applet_t234_sigheader.bin.encrypt
[   0.2419 ] BL: version 0.28.0.0-t234-54845784-ec016368 last_boot_error: 0
[   0.4006 ] Sending applet
[   0.5148 ] completed
[   0.5157 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.5165 ] MB2 Applet version 01.00.0000
[   0.7022 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.7031 ] MB2 Applet version 01.00.0000
[   0.7339 ] Retrieving board information
[   0.7347 ] tegrarcm_v2 --chip 0x23 0 --oem platformdetails chip chip_info.bin
[   0.7357 ] MB2 Applet version 01.00.0000
[   0.7814 ] Saved platform info in chip_info.bin
[   0.7869 ] Chip minor revision: 1
[   0.7871 ] Bootrom revision: 0x7
[   0.7873 ] Ram code: 0x0
[   0.7874 ] Chip sku: 0xd2
[   0.7877 ] Chip Sample: prod
[   0.7877 ] 
[   0.7881 ] Retrieving EEPROM data
[   0.7882 ] tegrarcm_v2 --oem platformdetails eeprom cvm /home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/cvm.bin --chip 0x23 0
[   0.7892 ] MB2 Applet version 01.00.0000
[   0.8371 ] Saved platform info in /home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/cvm.bin
[   0.8710 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.8717 ] MB2 Applet version 01.00.0000
[   0.9021 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.9028 ] MB2 Applet version 01.00.0000
[   0.9531 ] Dumping customer Info
[   0.9537 ] tegrarcm_v2 --chip 0x23 0 --oem dump bct tmp.bct
[   0.9540 ] MB2 Applet version 01.00.0000
[   1.0018 ] Saved bct in tmp.bct
[   1.0105 ] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/custinfo_out.bin
[   1.0114 ] Custome[   1.0121 ] r data saved in /home/labuser/flashing/images/june-29-2023/L4T_35.2.1/Linux_for_Tegra/bootloader/custinfo_out.bin successfully
[   1.0122 ] Rebooting to recovery mode
[   1.0131 ] tegrarcm_v2 --chip 0x23 0 --ismb2
[   1.0462 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   1.0470 ] MB2 Applet version 01.00.0000
[   1.0932 ] Booting to recovery mode
[   1.0936 ] tegrarcm_v2 --chip 0x23 0 --reboot recovery
[   1.0938 ] MB2 Applet version 01.00.0000
Parsing board information failed.

We also wrote a script (see below) to calculate and compare the CRC to make sure it’s not an issue related to the EEPROM getting overwritten and we see this “Parsing board information failed” error message on units with the same CRC (i.e. confirmed that the EEPROM isn’t getting overwritten).

import subprocess

def AddToCRC(b, crc):
    b2 = b
    if (b < 0):
        b2 = b + 256
    for i in range(8):
        odd = ((b2^crc) & 1) == 1
        crc >>= 1
        b2 >>= 1
        if (odd):
            crc ^= 0x8C # This means crc ^= 140.
    return crc


def read_eeprom():
    crc = 0
    for i in range(255):
        cmd = f"sudo i2cget -y 0 0x50 {i}"
        result = subprocess.getoutput(cmd)
        crc = AddToCRC(int(result, 16), crc)
    # print("Calculated CRC: ", hex(crc))
    cmd = f"sudo i2cget -y 0 0x50 255"
    eeprom_crc = subprocess.getoutput(cmd)
    if eeprom_crc == hex(crc):
        print(f"Calculated CRC same as EEPROM: {eeprom_crc}")
    else:
        print("Fail! calculated crc and eeprom crc do not match!!")
        print(f"Written crc: {eeprom_crc} Calculated: {hex(crc)}")
    #print("Current CRC: ", eeprom_crc)

if __name__ == "__main__":
    read_eeprom()

Could someone please help us resolve this issue?

Hi,

so you are still able to boot into the system to check the EEPROM,
but re-flash fails?

Can you show your sudo i2cget -y 0 0x50 here?
Also, just to make sure, are you using the correct BSP from the vendor?

Yes, we are able to boot into the system because we were able to flash the SOM after multiple retries so something is intermittent and yes we are using the correct BSP from the vendor.

Here’s what I see on the EEPROM:

xx@jetson-afd8:~$ sudo i2cget -y 0 0x50
0x00
xx@jetson-afd8:~$ sudo i2cdump -y 0 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00    ................
10: 00 00 00 0a 36 39 39 2d 31 33 37 30 31 2d 30 30    ...?699-13701-00
20: 30 34 2d 35 30 30 20 46 2e 30 00 00 00 00 00 00    04-500 F.0......
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 d4 af 91 2d b0 48 31 34 32 32 38 32    ....???-?H142282
50: 32 30 36 31 35 39 36 00 00 00 00 00 00 00 00 00    2061596.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 00 00 4d 31 00 00    ......NVCB..M1..
a0: 00 00 00 00 00 00 00 00 00 00 00 00 d4 af 91 2d    ............???-
b0: b0 48 0a 00 00 00 00 00 00 00 00 00 00 00 00 00    ?H?.............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e4    ...............?

So is your issue solved now?

No, we still see this issue as it’s intermittent and need help figuring out the root cause. Is this EEPROM write protected? Should we not have other devices connected to I2C bus 0 even if they are on different I2C addresses? Are you able to share the part number of the EEPROM so that we can look at the datasheet? Perhaps we are not meeting the rise time specification?

Hi,

is this the full log you can get when the device failed to be flashed?
If the issue is intermittent/happens occasionally, then it’s more like some USB communication issues between the device and the host,
because if it’s about the EEPROM, the flashing process is 100% guaranteed to fail.

Yes this is the full log message when the device fails to flash. When you say “USB communication issues between the device and the host” are you referring to an issue with the HW (e.g. USB cable, CTI’s Forge carrier board, or Orin) or the USB driver (on the host or device side)? Also, why would the log message consistently show “parsing board information failed” if this is related to an issue with USB communication? Is this a catch all error message for any issues related to the EEPROM and USB communication?

Hi,

I’m just saying maybe here. You have to know that it’s hard to debug a issue if it cannot be re-produced consistently.

For your case, because you are sure that the content of EEPROM is correct, so I think the only reason to this error is because the host fails to even read contents out of it, not to mention checking the correctness.

Understood. That’s part of the reason why I think there’s something wrong with the I2C bus 0 - my understanding is that’s how the SOM/Orin reads and validates the content of EEPROM, not via USB. We’ve also seen this intermittent issue on multiple SOM + CTI forge carrier board combination so it seems like this is due to some design flaw (either on the carrier board or SOM). The big assumption here is that the flashing script (flash.sh) is stable.

We also tried setting the “SKIP_EEPROM_CHECK” in flash.sh (as shown below) and this seems to work i.e. the flashing process is always successful with this flag enabled. This further confirms that there’s some issue related to the I2C bus. We obviously can’t skip the EEPROM check as we rely on this process to get the MAC address of the board. Do you know any other workaround or can you suggest anything else we can do to further troubleshoot this issue? We’ve also brought up this issue to CTI (i.e. the company that designed the forge carrier board) and they think this is an issue related to the SOM/Orin and/or the flashing script.

        if [ "${SKIP_EEPROM_CHECK}" = "" ]; then
                boardid=$(./chkbdinfo -i cvm.bin);
                boardFAB=$(./chkbdinfo -f cvm.bin);
                boardsku=$(./chkbdinfo -k cvm.bin);
                boardrevision=$(./chkbdinfo -r cvm.bin);
                chkerr "Parsing board information failed.";
        fi;

Hi,

I mean it’s right that SoC reads out contents of EEPROM via I2C, but it relies on USB to send the data to your host and do validation, as you can see with the command ./chkbdinfo -x cvm.bin. Also, if the issue arises from I2C bus, this contradicts with the fact that you can still run sudo i2cget/i2cdump -y 0 0x50 after you get into the system.

Yes we can run i2cdump but that doesn’t necessarily mean the i2c bus is stable. The reason why we think this is related to the I2C bus is because we see that flashing is successful as soon as we disconnect a cable connected to P17 on the forge carrier board (https://connecttech.com/ftp/pdf/CTIM-AGX201_Manual.pdf) which has connections to I2C bus 0. We conducted this test several times today and this seems to be repeatable. We also tried using different and shorter USB cables today for USB OTG (and USB debug to be consistent) and that didn’t help.

Hi,

your analysis sounds reasonable.
May I ask what kind of device was attached to the P17 on your carrier board?
Also, can you find a DevKit for cross validation?

Hi Dave,

We have a custom board connected to P17 that have various I2C and SPI devices. This board also provides the main power to the carrier board. We’ll try a devkit, but I think it’ll mostly be good for validating the BSP and flashing script which I think are stable based on our findings so far. What we also noticed is that we see bad data on the I2C bus when we are unable to flash and the SOM is in forced recovery mode and the I2C bus is stable when we are able to successfully flash and boot Linux.

Also, is there a way to write protect the EEPROM on the SOM?

Hi,

Documents of the Forge Carrier board states that the P17 is connected to I2C bus 1 and 2, but you said it’s connected to bus 0. Can you confirm that? EEPROM is at bus 2, so if the documents is right, then it’s indeed that data at I2C bus 2 will interfere with the flashing process, and we may need to check if we need to prevent it.

No, you can’t do that.

Hi,

I just checked with our hardware team, and they suggested that it’d be better to prevent connecting peripherals that is at the same I2C bus as EEPROM.

Hi Dave,

I can confirm that I2C1 (as noted in CTI’s Forge manual) is connected to bus 0 and I2C2 is connected to bus 7. You can see that I read the content of the EEPROM using the command shown below:

Just out of curiosity, why is the EEPROM not write protected? The content of this EEPROM (such as the MAC address) is critical to system (especially for the flashing process) so shouldn’t it be write protected?

Hi,

there are still sections in the EEPROM that user can define themselves (like the alternative MAC address), so we don’t set it as write-protected.

Hi Dave,

We are also seeing that the EEPROM on some SOMs are getting corrupted as shown below:

mfg@jetson-6f59:~$ sudo i2cdump -y 0 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 02 00 fe 00 00 00 00 00 00 00 00 00 00 00 00 00    ?.?.............
10: 00 00 00 0a 36 39 39 2d 31 33 37 30 31 2d 30 30    ...?699-13701-00
20: 30 34 2d 35 30 30 20 47 2e 30 00 00 00 00 00 00    04-500 G.0......
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 55 6f 94 2d b0 48 31 34 32 32 38 32    ....Uo?-?H142282
50: 32 31 34 31 33 37 36 00 00 00 00 00 00 00 00 00    2141376.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 c2 00 00 4d 31 00 00    ......NVC?..M1..
a0: 00 00 00 00 00 00 00 00 00 00 00 00 55 6f 94 2d    ............Uo?-
b0: b0 48 0a 00 00 00 00 00 00 00 00 00 00 00 00 00    ?H?.............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c3    ...............?
mfg@jetson-6f59:~$ python crc_check.py 
Fail! calculated crc and eeprom crc do not match!!
Written crc: 0xc3 Calculated: 0x31
mfg@jetson-6f59:~$ 

The corruption most likely happened during the flashing process because we don’t do anything to the device until eqos0 is available (this is how we SSH into the device). We tried manually bringing the device online through the serial debug port but it couldn’t find eqos0 as shown below:

mfg@jetson-6f59:~$ sudo ip link set up dev eqos0
Cannot find device "eqos0"
mfg@jetson-6f59:~$ 

Have anyone else reported this issue? We’ve seen this issue on a few devices now.

Hi,

Did that happen when P17 was connected?
If not, please file another topic as it does not seem to be related to your original issue.