Hi Xu_Xu,
Are you using the devkit or custom board for AGX Xavier?
It seems the similar issue as following topic.
Orin NX SPI delay between transfers - Jetson & Embedded Systems / Jetson Orin NX - NVIDIA Developer Forums
You could try to port the driver from K4.9 to K5.10 to check the performance.