SD card damage protection

Hi,

I’m new to this forum, so please forgive me if I’m on the wrong branch.

I wonder if an SD card in a Jetson Nano can be damaged as well by sudden power loss or crashes, etc., as is known from the Raspberry PI.

In all my RPI projects I could work around this problem quite well by making the filesystem read-only. While some claim this does not give 100% protection (which I believe as well), in many installations and even more tests I have not encountered a corrupted SD again.

My questions now:

  1. Does this problem (SD card damaging) exist with the Jetson Nano (I fear yes)?
  2. If so, would an eMMC get around the problem?
  3. If so, would a read-only filesystem be possible alternatively and is there already corresponding preliminary work e.g. gists, how to build something like that?

Thanks in advance for your answers.

Regards

  • Any filesystem (software) which is mounted with write ability can have the filesystem damaged by sudden power loss, including SD cards.
  • Any computer hardware subjected to power spikes can be destroyed by the spikes.
  • Any filesystem which is mounted read-only in Linux should never have an issue with the filesystem software.
  • All eMMC and SD card software and hardware will have the same exact answer as to harm of hardware or software relative to spikes and filesystem damage.
  • A filesystem which is read-write, but not used, could be remounted read-only.
  • OverlayFS would be read-only from the very start, and any write would go to RAM as an overlay on top of the actual read-only content, and thus give the illusion of being writable…the RAM portion has no protection and will go away each boot, but the read-only portion should be guaranteed to not be harmed so long as hardware is not harmed.
  • OverlayFS may require you to build a custom initial ramdisk. People have asked about this before, and it should probably work, but it might have a learning curve.
  • If you have a spare unused partition, and want to mount it read-only, then this trivial and easy. When you start making the actual operating system read-only you can expect it to be much more complicated.

Thanks for the comprehensive answer. Didn’t expect eMMC to be vulnerable like SD, but thinking twice it makes sense.
OK, so then I will try to apply my procedures known from the PI.

Thanks

Well, I tried and tried and found so many others, who did the same w/o success or w/o finally sharing their success.

  • OverlayFS would be read-only from the very start, and any write would go to RAM as an overlay on top of the actual read-only content, and thus give the illusion of being writable…the RAM portion has no protection and will go away each boot, but the read-only portion should be guaranteed to not be harmed so long as hardware is not harmed.

This seems to be the way to go, but I can’t make it work. Whatever I try, in the end the box is just endlessly booting :(

It is somehow unclear to me, why there is no such a thing like a tutorial to follow. It is not a weird request to have the FS protected, especially for kiosk applications. But seems, nobody is using a Nano like so :(

I have not personally installed OverlayFS, but I would consider looking closely at this:

  • Whether the initial ramdisk needs changes.
  • Whether options in the device tree’s “chosen->bootargs” needs to be adjusted.

I’m sure it won’t be easy since Jetson boot flow is custom (like most embedded systems), and probably “more custom” than other embedded systems.

Thanks, @linuxdev, I appreciate your efforts. Unfortunately I’m not the Linux Pro to have a minimal understanding of what you mean :)

This still won’t answer all you need, but will probably be useful…

When a system boots the bootloader itself is an operating system. If the bootloader needs content in a partition, then the bootloader itself needs the driver for that access. In the case of accessing configuration files on an ext4 filesystem, then the bootloader would also need ext4 drivers.

At the moment the Linux kernel loads the bootloader has essentially killed itself off by overwriting with a new operating system. As this occurs the Linux kernel needs its own drivers for reading partitions and for reading ext4. This is rarely a problem.

The trick is that some drivers may not be part of the Linux kernel Image file. Some drivers might be in the form of a module. Those modules are located somewhere else. Let’s say that you are working on some weird advanced filesystem which the default kernel does not know about, and that the code is only available in the form of a module. But the module itself is in that weird filesystem type. A bit of a “chicken and the egg” dilemma. To read the module you would need to load the module…but you can’t load the module until you read the module.

Thus, although you could try to boot straight to your normal boot environment, there is instead a bit of an adapter known as the “initial ramdisk”, a.k.a., the “initrd”. Every kernel knows how to boot to a limited self-contained boot environment running purely in RAM and without any filesystem type at all. It is literally just a gzip’d list of files that look like a tiny Linux installation.

That installation is good because no filesystem driver is needed. If you’ve copied the minimal set of drivers to read your advanced experimental filesystem to the initrd, then it means this temporary ramdisk system can load the module. The init (the first program a kernel executes other than itself) of the initrd does nothing more than basically the same thing the bootloader does: It reads everything needed and then overwrites itself again, but this time with the actual full Linux operating system after having loaded the module which provides for your test filesystem type. The initrd is an adapter for cases when required boot drivers are in the form of a module and the module is not normally reachable without extra effort.

I don’t know, but OverlayFS probably needs some such information available to it within an initrd, but can’t say for certain. If the main kernel is unable to load the required OverlayFS code, then it would be typical to use an initrd first, and use the initrd to hand off to the main Linux kernel. This might not be needed, but it is the first thing I examine when I can’t boot correctly due to a non-conventional filesystem issue.

If you look at “/boot/extlinux/extlinux.conf”, and if the boot entry has an “INITRD” key/value pair naming an image, then it means your system is starting with an initrd and not just booting directly to your expected kernel. If there is an INITRD entry, then you might look at the OverlayFS documentation and see what it needs for use in an initrd. If you get specific questions, then probably someone here can answer.

Thanks again for your patience. Unfortunately I have bricked my Nano yesterday evening while flashing it using the Nvidia SDK Manager (sigh!). Once it is replaced (not decided yet, since I revived my RPI4/Coral TPU approach today and it works comparable and I’m way more confident with RPI than with Nano, especially with the RO things) I will consider to return to this. Bookmarked! Thanks

BTW: I remember to have seen this “initrd” entry in the bootconfig. I was using the JP45 image provided by Nvidia.

So long as the carrier board is the dev kit one, then that image should be good. In the case of a third party carrier board, then the initrd would (possibly) need to change.

I also tried to establish an overlay fs using the steps other’s have outlined her. In the end I could not achieve to see a real RO rootfs. Files created on a rootfs, which was supposed to be RO did survive power cycles, that’s what I could achieve. Not really RO enough :)

Could you get a full serial console boot log? Maybe something would be notable.

I need to wait for a replacement of the box. Will let you know.

@linuxdev: I got a replacement board and set it up. It would now be ready to made read-only. If you are still motivated, I would like to ask you to resume the guidance.

This time during the in initial setup of a 4.5 pre-canned SD image (downloaded from Nvidia) I got this notice (not sure if it matters in the following attempts):

I’m prepared to do everything to achieve a RO rootfs and maybe finally this walk-through helps others too, which are desperately looking for a way to achieve this.

This is my /boot/extlinux/extlinux.conf

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 

# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
#      sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot

# LABEL backup
#    MENU LABEL backup kernel
#    LINUX /boot/Image.backup
#    INITRD /boot/initrd
#    APPEND ${cbootargs}

OK, I would like to report, what I did so far and see, if that will lead to something.

  • I was flashing a new image as already shown from a downloaded JP45 image
  • Booted it up and configured it, so that it at least did run
  • Then I applied a couple of instructions I took from an older gist of mine in order to make /rootfs RO. That gist has been run on an RPI and it works like a champ. It makes the entire SD RO and redirects all other system writes to /tmp

I would like to share that here, maybe somebody else is adventurous enough to follow. Be warned: This is work in progress and does not finally boot w/o probs. I have removed all the parts, which definitely only apply to the RPI.

In the end the boot process is stuck at Snap Daemon startup, but it already has problems on the way to there.

At least I could open another console and login. The rootfs is RO, but the system is barely operational.

The entire story and all commands so far here: https://gist.github.com/neilyoung/5a680052dbdce6258183754031c023e9

But I think it will not work finally, since I will most likely not be able to redirect all write attempts to the root-fs like so…

Other than the rootfs partition the other partitions are needed only during boot. The arrangement of where those partitions are stored has change, and it is telling you that if you continue, then some of the older partitions which are now in QSPI storage will be deleted. That’s ok since you won’t be booting multiple releases on a single SD card anyway. An exception might be if you are using multiple SD cards and some have older layouts while others use the QSPI layout…some older SD cards would then fail without a flash. Are you in need of booting both older and newer releases on the same Jetson? If not, then don’t worry.

FYI, I advise removing the “quiet” from all lines in extlinux.conf unless you are actually shipping a product and don’t want to see logs. Also, the “root=/dev/mmcblk0p1” is incorrect unless you want to boot to eMMC. It would find this on the SD card and then migrate back to eMMC for an eMMC model. If the SD card is mmcblk1, then you’d edit to “root=/dev/mmcblk1p1”. Better yet, have two entries, leave the one you have as is (other than removing “quiet”), and add a second entry which uses “root=/dev/mmcblk1p1”, and pick that via serial console at boot time.

If that is set up and tested, then perhaps a new initrd is required to add the OverlayFS is that is what you want to do. Just running purely read-only will fail since many apps (including the operating system) need temporary files.

Are you in need of booting both older and newer releases on the same Jetson? If not, then don’t worry.

Nope :)

FYI, I advise removing the “ quiet ” from all lines in extlinux.conf unless you are actually shipping a product and don’t want to see logs. Also, the “ root=/dev/mmcblk0p1 ” is incorrect unless you want to boot to eMMC . It would find this on the SD card and then migrate back to eMMC for an eMMC model. If the SD card is mmcblk1 , then you’d edit to “ root=/dev/mmcblk1p1 ”. Better yet, have two entries, leave the one you have as is (other than removing “ quiet ”), and add a second entry which uses “ root=/dev/mmcblk1p1 ”, and pick that via serial console at boot time.

OK, but this is the original extlinux.conf, I didn’t change it.

What I noticed while checking the board (I got today): I found this “X” holder (not sure if I express that right), under which the eMMC is. Maybe they have sent me an eMMC version? Not sure.

But the fun fact is: I need to return this box again (!!) The SD caddy is not holding the cards anymore after some in and out for flashing…

So yet another week to wait for the next replacement…

If that is set up and tested, then perhaps a new initrd is required to add the OverlayFS is that is what you want to do. Just running purely read-only will fail since many apps (including the operating system) need temporary files.

I’m not sure with this. I tried that already earlier

sudo update-initramfs -c -k $(uname -r)

It produced an image which I already found in the system. But even if I added this as RAM disk, the system came up, but it wasn’t really read-only. I stored a file on /rootfs, power-cycled the box and the file was still there…

If you are changing rootfs, then you must change this. Not sure about if this were an eMMC module or not, but dev kits come with a carrier, eMMC modules do not.

Not sure the update-initramfs will do the right thing. It may need manual setup and creation.

If you are changing rootfs, then you must change this.

Hmm. You mean, if I apply all my attempts to make it read only as described in the gist? Not sure I understand this sentence…

Not sure about if this were an eMMC module or not, but dev kits come with a carrier, eMMC modules do not.

I don’t remember to have seen this “cross holder” (5:42 in this video https://www.youtube.com/watch?v=xE1Jk8hHjNE) on my former board, but I can be wrong. The last board had such a thing.

Any commands you give to mount a device which are passed to the kernel at the moment the kernel loads need to be edited in extlinux.conf as well. Let’s say you use a different boot device…if you did not edit extlinux.conf, then it will still point at the old device. Or if you pass an argument somewhere else for rootfs modes, but failed to change it in extlinux.conf, then it might still use the same arguments as it did before and ignore where you think the arguments are coming from.

If for example extlinux.conf points at the SD card for boot, and the boot device providing extlinux.conf changes, but the content is the same, then you won’t be using anything on the new device other than an exact copy of the original exltinux.conf…which would know nothing about your changes.

Any kind of mounting stiffener will be irrelevant. If this is a module by itself, as shipped, then it will be an eMMC model and not a dev kit. If the Nano arrived on a carrier board, then it will use an SD card to boot and will be a dev kit model.

If you boot such that it points at