I managed to downgrade the bootloader when flashing 5.1.1 (rev. 1) release using sdkmanager. After that the version numbers were correct in the system so that I could upgrade the bootloader again using release 5.1.3. After that the bootloader could be upgraded again using nvidia-l4t-jetson-orin-nano-qspi-update.
My issue is now resolved so this thread can be closed.