To further investigate the issue, it is necessary to build a debug UEFI binary.
The problem appears to be specific to your custom carrier board, as we have not observed the same delay on the AGX Orin devkit.
You would likely have the best understanding of the design differences between your board and the reference devkit.