There is also memory on the Nano module. That memory works for a certain range of SD card versions. If your SD works with what is on the module’s QSPI memory, then it should “just work”. If not, then you have to flash the module (the SD card would not need to be touched if the Nano was already flashed with a QSPI release compatible with the 4.5.1 SD card).
FYI, on the Ubuntu 18.04 host PC with JetPack/SDK Manager installed you could pick a release version by starting this way on command line:
sdkmanager --archivedversions