I found another post dealing with a similar problem: [AGX Orin Recovery Boot - #5 by user117897]. The fix is to do this:
“Press ESC to enter UEFI Menu, then choose Device Manager → NVIDIA Configuration → L4T Configuration → OS chain A status → (The value is Unbootable if UEFI attemps recovery kernel) choose Normal → Save and exit, reboot, UEFI will try Direct Boot.”
I believe that the boot process gives up after some failed attempts, hence one has to manually force it to not give up after going back to the original SSD.
So the problem is solved.