Hi xllin,
Are you using the devkit or custom board for AGX Orin?
What’s the Jetpack version in use?
Could you check if the following threads could help for your case?
Orin NX SPI delay between transfers - #21 by KevinFFF
SPI TX Time Variability - #9 by francisco.silverio