Drive AGX Orin SoC error on restart

DRIVE OS Version: 6.0.10

Issue Description: Powering down and restarting the Drive AGX Orin results in failure to boot; reports SoC error and MCU logging a VRS11 error.

Power on the system
Command Executed
INFO: NvMCU_OrinTMON: Thermal Monitoring Enabled
I:RptrID - 0x810C PsCd - 0x10 - TMON All thermal events are clear
I:RptrID - 0x810C PsCd - 0x11 - TMON All alert events are clear
INFO: NvMCU_OrinTMON: Thermal Alert Trigger not present anymore
I:RptrID - 0x810C PsCd - 0xF - TMON No notification present
INFO: MCU_ERRHANDLER: Published Power State: Power-up complete
INFO : MCU_ISTMGR: IST_DONE pin no monitor time of 2000 ms over.
I:RptrID - 0x810A PsCd - 0xD - ISTManager:IST DONE pin stuck at low not observed.
MCU_FOH: SOC error pin is asserted
ERROR: MCU_ERRHANDLER: SOC error pin is asserted
MCU_FOH: Spi Transmit Started
INFO: SftyMon_IoHwAbs: PG_VRS11 monitoring started...
ERROR: SftyMon_IoHwAbs: VRS11 PGOOD Failure notification...!
ERROR: MCU_ERRHANDLER: SftyMon : ReporterID - 0x8110 ErrorCode - 0x1A 
 ERROR: MCU_ERRHANDLER: PGOOD failure notification...error detected by VRS11
INFO: MCU_SWC_FanControl: max_rpm fan2 : 0
INFO: MCU_SWC_FanControl: maxrpm of fan2 has more than 50 percent deviation against rated maxrpm

So far the only way I have been able to clear this is to powerdown and turn off the power on the back panel and wait a long time.

After the physical power-off and wait (~5-10min) the startup occurs normally enough:

*************** NvShell Initialization Start******************
DRIVE-V6.0.9-P3710-AFW-Aurix-StepB-5.14.15
Compilation date: Mar 13 2024, 20:44:10
Enter 'help' to see the available commands.

*************** NvShell Initialized  *************************
 Press 'Enter' for NvShell prompt 
*************************************************************


ERROR: MCU_PLTFPWRMGR: WDT reset (or aurixreset force) triggered...
I:RptrID - 0x810A PsCd - 0xF - ISTManager:IST execution status read success.
I:RptrID - 0x810A PsCd - 0xE - ISTManager:No plausibility error in NVM data.
I:RptrID - 0x810A PsCd - 0x14 - ISTManager:No range plausibility error in NVM data.
I:RptrID - 0x810D PsCd - 0x16 - PwrCtrl Init Ok
I:RptrID - 0x810F PsCd - 0x7 - BootchainCfg Cfg Range Check Success
I:RptrID - 0x810F PsCd - 0x7 - BootchainCfg Cfg Range Check Success
I:RptrID - 0x810F PsCd - 0x2 - BootchainCfg NvM Read Success
I:RptrID - 0x810F PsCd - 0x4 - BootchainCfg NvM Plaus Check Success
MCU_FOH: E2E Initialized
MCU_FOH: Initialization done
INFO: MCU_PLTFPWRMGR: Powering up 
INFO: MCU_PLTFPWRMGR: Voltage Monitoring is enabled.
INFO: MCU_PLTFPWRMGR: Send command 'voltageMonitor disable' on console to disable Voltage Monitoring.
INFO: MCU_PLTFPWRMGR: Monitoring for KL30 (G3CH4) started.
INFO: MCU_PLTFPWRMGR: KL30 (G3CH4) Voltage (14562mV) exceeded threshold (6700mV). Continuing...!
I:RptrID - 0x810D PsCd - 0x8 - PwrCtrl Dio Chk Ok
I:RptrID - 0x810D PsCd - 0x17 - PwrCtrl PrePwrUp Ok
INFO: PLTFPWRMGR_IOHWABS: Monitoring for PREREG_5V (G2CH2) started.
INFO: PLTFPWRMGR_IOHWABS: PREREG_5V (G2CH2) Voltage (5010mV) exceeded threshold (4850mV). Continuing...
INFO: MCU_PLTFPWRMGR: Monitoring for PREREG_SENSE_16V (G3CH6) started.
INFO: MCU_PLTFPWRMGR: PREREG_SENSE_16V (G3CH6) Voltage (12146mV) exceeded threshold (6700mV). Continuing...!
INFO: MCU_PLTFPWRMGR: Request Power-up Ethernet Switch done !
INFO: MCU_PltfPwrMgr: Linkup status is not active
INFO: MCU_PltfPwrMgr: 88Q5072 OAK Link Active
INFO:Marvell switch: 0: NW_CFG_BASE
INFO:Marvell switch: Marvell Oak/Spruce configuration completed..
INFO: MCU_PltfPwrMgr: Switch Init
INFO: Marvell Phy: Phy init begin..
INFO: Marvell Phy: Phy init completed..
INFO: MCU_PltfPwrMgr: PHYs Init
INFO: MCU_PltfPwrMgr: StbM Init
INFO: MCU_PLTFPWRMGR: Request Eth initialization done !
I:RptrID - 0x810B PsCd - 0x20 - VMON Config Valid
I:RptrID - 0x810B PsCd - 0x13 - VMON i2c addr load passed
I:RptrID - 0x810B PsCd - 0xA - VMON Bist Passed
INFO: NvMCU_OrinTMON: Orin-A Temperature sensor initialized 
I:RptrID - 0x810C PsCd - 0x17 - TMON Configuration in default mode pass
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0x3 - PwrCtrl I2C Read Ok
I:RptrID - 0x810D PsCd - 0x6 - PwrCtrl I2C Read Crc Ok
I:RptrID - 0x810D PsCd - 0x7 - PwrCtrl Config Ok
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0xD - PwrCtrl ActShdn Ok
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0xE - PwrCtrl Actslp Ok
Check for VRS10............................................................
Check for VRS10............................................................
Check for VRS10............................................................
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0x9 - PwrCtrl Vrs10 Int Ok
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0x2 - PwrCtrl I2C Write Ok
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0x4 - PwrCtrl I2C Val Ok
Check for VRS11-1..........................................................
Check for VRS11-2...........................................................
Check for VRS11-1..........................................................
I:RptrID - 0x810D PsCd - 0xA - PwrCtrl VRS11 Int Ok
Check for VRS11-2...........................................................
I:RptrID - 0x810D PsCd - 0xA - PwrCtrl VRS11 Int Ok
Check for VRS11-1..........................................................
Check for VRS11-1..........................................................
Check for VRS11-1..........................................................
Check for VRS11-1..........................................................
Check for VRS11-2...........................................................
Check for VRS11-2...........................................................
Check for VRS11-2...........................................................
Check for VRS11-2...........................................................
INFO: NVMCU_ORINPWRCTRL: Programming thermal thresholds done!
Check for VRS10............................................................
I:RptrID - 0x810D PsCd - 0xC - PwrCtrl Actshdn Ok
Check for VRS10............................................................
INFO: NVMCU_ORINPWRCTRL: FUNC_NIRQ continuous monitoring Enabled!
INFO: SftyMon_IoHwAbs: toggle check of local and remote sensor successfull
INFO: NvMCU_OrinTMON: toggle check of local and remote sensor successfull
I:RptrID - 0x810C PsCd - 0xB - TMON Toggle Check for Alert & Shutdown Pass
INFO: NvMCU_OrinTMON: Orin-A Temperature sensor initialized 
I:RptrID - 0x810C PsCd - 0x17 - TMON Configuration in default mode pass
I:RptrID - 0x810B PsCd - 0x8 - VMON Seq Record
I:RptrID - 0x810B PsCd - 0x7 - VMON Seq Match
INFO: SftyMon_IoHwAbs: Board Temperature sensor initialized
INFO: NvMCU_OrinTMON: Orin TMON init complete, post toggle check
I:RptrID - 0x810B PsCd - 0x23 - VMON Seq Ackld
I:RptrID - 0x810B PsCd - 0xD - VMON Plausibility pass
INFO: MCU_PLTFPWRMGR: Orin TMON enabled .... 
INFO: MCU_PLTFPWRMGR: Board TMON enabled.
INFO: BtChn_Cfg: Tegra x1 Boot Chain is : B 
INFO: BtChn_Cfg: Btchn Pin ORIN_BOOTCHAIN0 set to 1
INFO: BtChn_Cfg: Btchn Pin ORIN_BOOTCHAIN1 set to 0
I:RptrID - 0x810F PsCd - 0x6 - BootchainCfg Dio set success
Check for VRS10............................................................
Check for VRS10............................................................
Check for VRS10............................................................
INFO: NVMCU_ORINPWRCTRL: Tegra reset released
MCU_FOH: MCU FOH : Initiate SOC Error Pin Monitoring & SPI communication
MCU_FOH: MCU FOH : Start Monitoring Initiated
INFO: MCU_PLTFPWRMGR: Power-up sequence is complete !
INFO: CmnIf: Wdg Enabled Success!
INFO: MCU_PLTFPWRMGR: DL Ready drive to HIGH!
Power on the system
INFO: NvMCU_OrinTMON: Thermal Monitoring Enabled
I:RptrID - 0x810C PsCd - 0x10 - TMON All thermal events are clear
I:RptrID - 0x810C PsCd - 0x11 - TMON All alert events are clear
INFO: NvMCU_OrinTMON: Thermal Alert Trigger not present anymore
I:RptrID - 0x810C PsCd - 0xF - TMON No notification present
INFO: MCU_ERRHANDLER: Published Power State: Power-up complete
INFO : MCU_ISTMGR: IST_DONE pin no monitor time of 2000 ms over.
I:RptrID - 0x810A PsCd - 0xD - ISTManager:IST DONE pin stuck at low not observed.
MCU_FOH: Spi Transmit Started
MCU_FOH: SOC error pin is de-asserted
ERROR: MCU_ERRHANDLER: SOC error pin is de-asserted
MCU_FOH: Periodic Status [0] 0xab [1] 0xcd [250] 0x12 [251] 0x34
INFO: SftyMon_IoHwAbs: PG_VRS11 monitoring started...
INFO : MCU_ISTMGR: IST Manager initialized to send/receive commands 
I:RptrID - 0x810A PsCd - 0x11 - ISTManager:IST result fetched.
INFO : IST_TESTAPP: IST Result Ready to fetch
INFO: MCU_SWC_FanControl: max_rpm fan2 : 0
INFO: MCU_SWC_FanControl: maxrpm of fan2 has more than 50 percent deviation against rated maxrpm
MCU_FOH: SOC error pin is asserted
ERROR: MCU_ERRHANDLER: SOC error pin is asserted
MCU_FOH: ErrReport: ErrorCode-0x21ed ReporterId-0xe008 Error_Attribute-0x0 Timestamp-0x214ffec5

I have reviewed other related posts with similar or adjacent symptoms and have not been able to ascertain how to remedy this.

This has blocked automatic testing which requires the Drive AGX Orin to restart. A tedious and timeconsuming manual intervention is required on each reboot.

Is this expected (yet unfortunate) behavior?

Dear @david.cattley ,
How did you reboot the machine? Did you try common_if_testapp -mcureset`?

yes that is excactly how this is triggered in my bench

the automation uses that exact command

if [[ ${REBOOT} -eq 1 ]]; then
	if ! command -v common_if_testapp >/dev/null 2>&1; then
		echo "${PROG}: common_if_testapp not found (required for --reboot)" >&2
		exit 1
	fi
	# Arm MCU first so its handshake window overlaps Linux halt teardown.
	# Do not pass -tDelay_mcuwait: Thor returns CommonIf_E_INVALID_PARAMS;
	# Orin documents it, but the common safe subset omits the delay.
	echo "${PROG}: arming MCU reset (common_if_testapp -mcureset)"
	common_if_testapp -mcureset
fi

sync
echo "${PROG}: shutting down (shutdown -h now)"
exec shutdown -h now

Dear @david.cattley ,
Does reflashing the target fixes the issue? I don’t see any issue with common_if_testapp -mcureset on my target.

Sorry for the delayed reply.

No. In fact it was after such a reflash that I first attempted using the common_if_testapp and hit the issue.

Could you update the MCU FW version and check if it resolves the issue?

Update it to what? That is the latest version of the FW provided in the 6.0.10 kit as best I can tell.
When this kit was first upgraded from 6.0.7 (or something) to 6.0.10 long ago the MCU FW was updated to this version as this is what is in the 6.0.10 SDK.

Is there some other source of Aurix FW for the Orin?

after forcing a reflash of the MCU and power-cycle/reboot:

repeated tests of common_test_app -mcureset now seem to be fine.

so resolved by forcing a reflash of the Aurix?