Orin Industrial wont boot on custom carrier

I am struggling to get the Orin Industrial module working on my custom carrier board.
It has been working fine with the Developer kit, 32GB and 64GB modules.
I’ve implemented the SYS_VIN_SV connection to AO_3V3 as required.

The module has been flashed and tested on a developer kit carrier, so I know its working.
At first, it wouldn’t do anything at all on my carrier board. After I saw this post and implemented the suggested fix, I at least started getting some output from MB1, but it gets stuck part of the way through.

My carrier board has a custom power sequencer on it and does not use the example circuit from the design guide. It is controlled from a MSP430 microcontroller so I can adjust some of the timings if necessary. SYS_VIN_HV and SYS_VIN_MV are driven from the same signal and come up at the same time.
I connected a logic analyzer to the board and captured some timing diagrams of the power sequence.
The channels are as follows:
D0 - SYS_RESET_N
D1 - VDDIN_PWR_BAD_N
D2 - Not used
D3 - CARRIER_POWER_ON
D4 - MODULE_SHDN_N
D5 - VIN_PWR_ON (enables SYS_VIN_HV and MV)
D6 - AO_3V3
CH1 - MODULE_POWER_ON

The timings look pretty much identical. The first picture is from the Industrial module and the second is from a developer kit module.


Here is the output from MB1 up to where it gets stuck

[0000.061] I> MB1 (version: 1.2.0.0-t234-54845784-562369e5)
[0000.066] I> t234-A01-1-Silicon (0x12347) Prod
[0000.071] I> Boot-mode : Coldboot
[0000.074] I> Entry timestamp: 0x00000000
[0000.077] I> last_boot_error: 0x0
[0000.081] I> BR-BCT: preprod_dev_sign: 0
[0000.084] I> rst_source: 0x0, rst_level: 0x0
[0000.088] I> Task: SE error check
[0000.092] I> Task: Bootchain select WAR set
[0000.096] I> Task: Enable SLCG
[0000.099] I> Task: CRC check
[0000.101] I> Skip FUSE records CRC check as records_integrity fuse is not burned
[0000.108] I> Task: Initialize MB2 params
[0000.113] I> MB2-params @ 0x40060000
[0000.116] I> Task: Crypto init
[0000.119] I> Task: Perform MB1 KAT tests
[0000.123] I> Task: NVRNG health check
[0000.127] I> NVRNG: Health check success
[0000.130] I> Task: MSS Bandwidth limiter settings for iGPU clients
[0000.136] I> Task: Enabling and initialization of Bandwidth limiter
[0000.142] I> No request to configure MBWT settings for any PC!
[0000.148] I> Task: Secure debug controls
[0000.152] I> Task: strap war set
[0000.155] I> Task: Initialize SOC Therm
[0000.159] I> Task: Program NV master stream id
[0000.163] I> Task: Verify boot mode
[0000.169] I> Task: Alias fuses
[0000.172] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0000.179] I> Task: Print SKU type
[0000.182] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x00000000
[0000.187] I> FUSE_OPT_GPC_DISABLE = 0x00000000
[0000.191] I> FUSE_OPT_TPC_DISABLE = 0x00000000
[0000.196] I> FUSE_OPT_DLA_DISABLE = 0x00000000
[0000.200] I> FUSE_OPT_PVA_DISABLE = 0x00000000
[0000.204] I> FUSE_OPT_NVENC_DISABLE = 0x00000000
[0000.209] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0000.213] I> FUSE_OPT_FSI_DISABLE = 0x00000000
[0000.217] I> FUSE_OPT_EMC_DISABLE = 0x00000000
[0000.222] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0000.226] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0000.230] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0000.234] I> FUSE_SKU_INFO_0 = 0x90
[0000.237] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS 
[0000.242] I> FUSE_PACKAGE_INFO_0 = 0x2
[0000.245] I> SKU: Prod
[0000.247] I> Task: Boost clocks
[0000.250] I> Initializing PLLC2 for AXI_CBB.
[0000.255] I> AXI_CBB : src = 35, divisor = 0
[0000.259] I> Task: Voltage monitor
[0000.262] I> VMON: Vmon re-calibration and fine tuning done
[0000.267] I> Task: UPHY init
[0000.272] I> HSIO UPHY init done
[0000.275] W> Skipping GBE UPHY config
[0000.279] I> Task: Boot device init
[0000.282] I> Boot_device: QSPI_FLASH instance: 0
[0000.287] I> Qspi clock source : pllc_out0
[0000.291] I> QSPI Flash: Macronix 64MB
[0000.295] I> QSPI-0l initialized successfully
[0000.299] I> Task: TSC init
[0000.302] I> Task: Load membct
[0000.305] I> RAM_CODE 0x4000441
[0000.308] I> Loading MEMBCT 
[0000.310] I> Slot: 0
[0000.312] I> Binary[1] block-3840 (partition size: 0x40000)
[0000.318] I> Binary name: MEM-BCT-1
[0000.321] I> Size of crypto header is 8192
[0000.325] I> Size of crypto header is 8192
[0000.329] I> strt_pg_num(3840) num_of_pgs(16) read_buf(0x40050000)
[0000.335] I> BCH of MEM-BCT-1 read from storage
[0000.340] I> BCH address is : 0x40050000
[0000.343] I> MEM-BCT-1 header integrity check is success
[0000.349] I> Binary magic in BCH component 1 is MEM1
[0000.353] I> component binary type is 1
[0000.357] I> strt_pg_num(3971) num_of_pgs(115) read_buf(0x40040000)
[0000.364] I> MEM-BCT-1 binary is read from storage
[0000.369] I> MEM-BCT-1 binary integrity check is success
[0000.374] I> Binary MEM-BCT-1 loaded successfully at 0x40040000 (0xe580)
[0000.381] I> RAM_CODE 0x4000441
[0000.386] I> RAM_CODE 0x4000441
[0000.390] I> Task: Load Page retirement list
[0000.394] I> Task: SDRAM params override
[0000.398] I> Task: Save mem-bct info
[0000.401] I> Task: Carveout allocate
[0000.404] I> RCM blob carveout will not be allocated
[0000.409] I> Update CCPLEX IST carveout from MB1-BCT
[0000.414] I> ECC region[0]: Start:0x80000000, End:0xe80000000
[0000.420] I> ECC region[1]: Start:0x0, End:0x0
[0000.424] I> ECC region[2]: Start:0x0, End:0x0
[0000.428] I> ECC region[3]: Start:0x0, End:0x0
[0000.432] I> ECC region[4]: Start:0x0, End:0x0
[0000.437] I> Non-ECC region[0]: Start:0x0, End:0x0
[0000.441] I> Non-ECC region[1]: Start:0x0, End:0x0
[0000.446] I> Non-ECC region[2]: Start:0x0, End:0x0
[0000.451] I> Non-ECC region[3]: Start:0x0, End:0x0
[0000.455] I> Non-ECC region[4]: Start:0x0, End:0x0
[0000.466] I> allocated(CO:44) base:0xe49800000 size:0x36800000 align: 0x100000
[0000.473] I> allocated(CO:31) base:0xe40000000 size:0x8000000 align: 0x8000000
[0000.480] I> allocated(CO:43) base:0xe3c000000 size:0x4000000 align: 0x200000
[0000.487] I> allocated(CO:39) base:0xe39e00000 size:0x2200000 align: 0x10000
[0000.494] I> allocated(CO:20) base:0xe36000000 size:0x2000000 align: 0x2000000
[0000.501] I> allocated(CO:24) base:0xe34000000 size:0x2000000 align: 0x2000000
[0000.508] I> allocated(CO:28) base:0xe32000000 size:0x2000000 align: 0x2000000
[0000.516] I> allocated(CO:29) base:0xe30000000 size:0x2000000 align: 0x2000000
[0000.523] I> allocated(CO:22) base:0xe48000000 size:0x1000000 align: 0x1000000
[0000.530] I> allocated(CO:41) base:0xe38e00000 size:0x1000000 align: 0x100000
[0000.537] I> allocated(CO:35) base:0xe38000000 size:0xe00000 align: 0x10000
[0000.544] I> allocated(CO:02) base:0xe49000000 size:0x800000 align: 0x800000
[0000.551] I> allocated(CO:03) base:0xe2f800000 size:0x800000 align: 0x800000
[0000.558] I> allocated(CO:06) base:0xe2f000000 size:0x800000 align: 0x800000
[0000.565] I> allocated(CO:56) base:0xe2e800000 size:0x800000 align: 0x200000
[0000.571] I> allocated(CO:07) base:0xe2e400000 size:0x400000 align: 0x400000
[0000.578] I> allocated(CO:33) base:0xe2e000000 size:0x400000 align: 0x200000
[0000.585] I> allocated(CO:23) base:0xe2de00000 size:0x200000 align: 0x200000
[0000.592] I> allocated(CO:01) base:0xe2dd00000 size:0x100000 align: 0x100000
[0000.599] I> allocated(CO:04) base:0xe2dc00000 size:0x100000 align: 0x100000
[0000.606] I> allocated(CO:05) base:0xe2db00000 size:0x100000 align: 0x100000
[0000.613] I> allocated(CO:08) base:0xe2da00000 size:0x100000 align: 0x100000
[0000.620] I> allocated(CO:09) base:0xe2d900000 size:0x100000 align: 0x100000
[0000.627] I> allocated(CO:15) base:0xe2d800000 size:0x100000 align: 0x100000
[0000.634] I> allocated(CO:17) base:0xe2d700000 size:0x100000 align: 0x100000
[0000.641] I> allocated(CO:27) base:0xe2d600000 size:0x100000 align: 0x100000
[0000.648] I> allocated(CO:42) base:0xe2d500000 size:0x100000 align: 0x100000
[0000.655] I> allocated(CO:54) base:0xe2d480000 size:0x80000 align: 0x80000
[0000.662] I> allocated(CO:34) base:0xe2d470000 size:0x10000 align: 0x10000
[0000.668] I> allocated(CO:72) base:0xe2d270000 size:0x200000 align: 0x10000
[0000.675] I> allocated(CO:47) base:0xe2ce00000 size:0x400000 align: 0x200000
[0000.682] I> allocated(CO:48) base:0xe2d250000 size:0x20000 align: 0x10000
[0000.689] I> allocated(CO:69) base:0xe2d230000 size:0x20000 align: 0x10000
[0000.696] I> allocated(CO:49) base:0xe2d220000 size:0x10000 align: 0x10000
[0000.702] I> allocated(CO:50) base:0xe2d210000 size:0x10000 align: 0x10000
[0000.709] I> allocated(CO:52) base:0xe2d200000 size:0x10000 align: 0x10000
[0000.716] I> NSDRAM base: 0x80000000, end: 0xe2d270000, size: 0xdad270000
[0000.723] I> Task: Thermal check
[0000.726] I> max_chip_limit = 125
[0000.729] I> min_chip_limit = -43
[0000.732] I> max temp read = 31
[0000.735] I> min temp read = 30
[0000.738] I> Task: Update FSI SCR with thermal fuse data
[0000.743] I> Task: Enable WDT 5th expiry
[0000.747] I> Task: I2C register
[0000.750] I> Task: Set I2C bus freq
[0000.753] I> Task: Reset FSI
[0000.756] I> Task: Pinmux init
[0000.759] I> Task: Prod config init
[0000.763] I> Task: Pad voltage init
[0000.766] I> Task: Prod init
[0000.769] I> Task: Program rst req config reg
[0000.773] I> Task: Common rail init
[0000.776] I> DONE: Thermal config
[0000.780] W> DEVICE_PROD: module = 13, instance = 4 not found in device prod.
[0000

I also captured the same section of MB1 from a working developer kit module and, it seems that the industrial module is getting stuck at the SOC rail config step.

[0000.784] W> DEVICE_PROD: module = 13, instance = 4 not found in device prod.
[0000.792] I> DONE: SOC rail config
[0000.796] W> PMIC_CONFIG: Rail: MEMIO rail config not found in MB1 BCT.
[0000.803] I> DONE: MEMIO rail config
[0000.806] I> DONE: GPU rail info
[0000.809] I> DONE: CV rail info
[0000.812] I> Task: Mem clock src
[0000.815] I> Task: Misc. board config
[0000.819] I> PMIC_CONFIG: Platform config not found in MB1 BCT.
(boot continues)
  1. Please compare your HV and MV sequence to that of devkit.
  2. Please check if your board will change below strapping pins status during power on.

    The module includes buffers on all but two to ensure the SoC strapping pins retain their strapped state regardless of connections on the carrier board. For the unbuffered strap pins, the carrier board must ensure these pins are not pulled or driven low or high during power-on to avoid affecting the strap levels.

HV and MV come up at the same time, but I was able to modify the board to add a 3ms delay.
Does the devkit have accessible probe points to see these signals?
Normal:


Modified:

Same behavior with or without the modification.

As for point 2, there are buffers on my carrier board to address that exact issue. It gave me trouble when bringing up the 64G module previously.

Why is there a step on MV? You may need to compare that with devkit or try more timing between HV and MV. No other clue since you had added buffer to strap pins.

I was able to improve the ramp up somewhat but it still doesnt work.

I did notice that on my schematic pin J62 is marked as reserved and not connected but on the reference carrier design it is grounded. Could this be the culprit? What is the function of this pin? When I designed the board this was not in the design guide, and it looks like it was added in April 2023.

Its function: J62 GND (TEMP_SHDN_EN_N) Enables buffered TEMP_THERM_N to trigger shutdown.

Please short it to GND with 0 ohm resistor. This might be root cause.