It gets much worse. I initially use the SD Image based installation method with my devkit. My devkit had older firmware on it from the factory (which I believe was dated 2023-02-17). To solve as second issue Nvidia Jetson Orin shows 6.3GB of RAM - #22 by theofficialgman I flashed over my device using SDKManager. This gave me firmware dated 2023-03-19 and solved the ram issue (now at 7.2GB).
HOWEVER, the performance is now in the worse state regardless of what I do (both exclusive fullscreen and windowed are bad at ~125FPS). So it looks to me like this latest firmware has a serious bug in it and the previous one was bugged partially.
Please upload the 2023-02-17 firmware here and provide instructions for a manual flash so that I may flash it back to my devkit. I do not have any way of returning to it now.