Orin HDMI only supports up to HDMI2.0 so it cannot be achieved to 8K@60.
Also, you need to modify the DCB as document said.
https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/SD/Kernel/DisplayConfigurationAndBringUp/OrinDisplayconfig.html?highlight=dcb
If you only change the hardware design, then it is still in DP mode.
BTW, p3509 board config in the BSP already supported HDMI mode. But only extra patch is needed.