Hi,
What we observe on Xavier developer kit is with 1-seond delay occasionally, and then it goes back to synchronization. The symptom on your device is that RTC time keeps drifting, which we don’t observe on Xavier developer kit.
If you need further help, please try to reproduce it on developer kit and share us the steps. So that we can have teams check and do investigation.