After checking the logs you provided, I found the boot up process are almost the same and only one difference about if ethernet could Link Up.
I could not reproduce this issue on Xavier NX devkit with l4t-r35.1.
So, do you follow the adaptation guide for your custom board?
Jetson Module Adaptation and Bring-Up: Jetson Xavier NX Series — Jetson Linux
Developer Guide 34.1 documentation (nvidia.com)
I’ve also found two topics about the similar issue as yours.
Please refer them to make sure your RGMII configuration (pinmux, dts) is right for ethernet phy.
Does Orin support 88E1512 PHY chip? - Jetson & Embedded Systems / Jetson AGX Orin - NVIDIA Developer Forums
RGMII with switch can’t work - Jetson & Embedded Systems / Jetson AGX Orin - NVIDIA Developer Forums