From your log, there are two significant delay both relating to Phy.
[20:12:38:278] PROGRESS CODE: V03040002 I0␍␊
[20:12:38:283] ␊
[20:13:05:217] SNP:PHY: PhyDxeInitialization () Failed to reset Phy␍␊
[20:13:35:496] ArmSbmrStatusCodeCallback: Failed to send IPMI command - Unsupported␍␊
[20:14:02:314] SNP:PHY: PhyDxeInitialization () Failed to reset Phy␍␊
There’s a similar issue as yours also on the custom carrier board.
Please refer that to check if it could help.
UEFI with RGMII is slow to boot - Jetson & Embedded Systems / Jetson AGX Orin - NVIDIA Developer Forums