The proper way to implement production read-only filesystem

Hi,
after reading dozens of threads on this forum about how to make a read-only SSD or SD card to make your device never break in production devices (say as a result of power cut while there is an SD card write operation going on), I understood that the NX dev kit is not the right way to do it in production devices. Looks like for that purpose, one should use the Xavier NX module.

Ok then, but I was still not able to find any information on how is that supposed to be implemented. Is there any blog post on that? Anything guide in the documentation? Maybe I just don’t know how to ask…

Thanks!
Jan

hello jan.juran,

may I know what’s the actual use-case, what’s the threat model of file system.
instead of read-only filesystem, how about refer to Jetson Security to enable the Disk Encryption?

BTW,
there’re some discussion threads, such as Topic 78540, Topic 49050, and Topic 191060 ​for making rootfs as read-only by using overlayfsroot.
you should dig into this thread, Topic 191060, which tested with JetPack-4.6, and it’ll need a simple fixes to switch from the default MODULES=most to MODULES=dep to workaround the initrd.img too big for the usage.

Hi,
Thanks for your response.

Not sure what this has to do with security. My concern is not about anyone stealing the data, but about the device not being able to boot or otherwise getting corrupted as a result of invalid disk write operation (say because of a power cut). That’s not a theoretical issue. It happened multiple times during testing.

The device is a camera feed that does some image processing and displays a live feed. When you turn it on, its supposed to just run and thats it. When suddenly turned off, next time it has to boot exactly the same way. It does not need to write anything to any drive, everything is preconfigured.

Will go over the links you shared, but last time I checked it seemed that there’s no overlayFS support on Jetson. Not a working one that is.

Thanks

hello jan.juran,

you should refer to Bootloader and Root filesystems redundancy regarding to your use-case.
it ensures that a workable Bootloader partition remains on boot storage, you may duplicate A/B slot, switching to workable slot if the crash happened.

I recognise the problem, very common.

I provide code to implement 2x approaches for that very purpose.

The first is this one:

That does however require an SSD. It implements a read-only mounted SSD with a memory overlayFS so that the operating system still works.

The second one is this one:

This is like the first one except that it also creates two additional partitons. One is indeeded to be kept read-only and remounted read-write for configuration updates. The second is a read-write partition but it’s forceabled fsck’ed during boot before mounting in order to avoid having the boot process hang.

If you are interested, I could re-implement the GitHub - hcfman/sbts-boot-from-SSD: One command installation of boot from SSD for Jetson nano, Xavier NX and Xavier AGX to create a version that provides a read-only mount over the original SD card and publish it.

Note: I’ve used this approach for the jetson series computers for at least two years and for raspberrypi’s for many more years, so it’s very stable. And solves exactly the problem you are trying to solve.

Cheers,
Kim

Hi, that’s great news! I will have a look at your projects.

I like working with SD cards because those are easily backed up and swapped and do not require any special equipment. Just a USB dongle.

I think I can get away with using SSDs for this project though - it should be able to boot fast anyway. You can easily create an image of that as well, right? You know, so that you can copy it over along with all the settings (including OS) to other devices.

I come from Raspberry Pi environment, where you would just develop something, then create a release image for that (I use Windows, so for me it’s Win32DiskImager) and then use that image to create multiple copies of the product.

Thanks again.
Jan

@kimv9rqv One question though. Any idea why these guys died trying what you have in a repo with just a few commits? At some point during my research I read through the whole thread (which is a loooong evening) and after doing so fell in despair…

Very glad it’s worked out for you. It’s inspiring me now to make a single SD version of sbts-base project at some time. I’m still trying to get people’s attention to try my projects as they are very new. I’m happy with every new person that tries them :-)

The approach I use as well is rather unique and people will not be very familiar with what I’m doing . Not really unique though, I was inspired by Pascal Sutter’s raspberrypi project (overlayFS), but I don’t think that anyone else has done this by enhanced it further to pivot the boot to SSD with the extra help to forcably do an fsck -y each boot before mounting the extra partitions. At least not as an open source project. The end result should be a highly resilient installation. The sbts-base approach is the most flexible as it provides two extra partitions, one for holding configuration that you keep read-only till you need to make changes and a read-write partition. But the most familiar project would be the sbts-boot-from-SSD project.

I also use a lot of raspberrypi’s. These are the plug computers that I used before the pi always had problems with disk corruption, even if you thought that no writes had occurred before the power cycle. Around 10 years ago I made a commercial product and every one of them was sent back with corruption on the SD card. I had to find a working solution fast. Since then I’ve always used the overlayFS approach.

This last weekend I also applied the approach to make a raspberrypi version of StalkedByTheState with the same disk structure that runs just on an SD card. However, it’s purpose was not to function as an inference engine but rather to use the I/O Control and home automation functions that it provides. Specifically the interfacing with Phidget InterfaceKit I/O controllers and the rfxtrx433 home automation transceiver.

Cheers,
Kim Hendrikse

Meanwhile there is a solution using the default overlayfs, also here in this forum (search for lehni)

Nice to hear others are busy with the overlayFS. The approach I use explicitly avoids creating a new initramfs image and uses the standard one that it boots on. Initially I was also building an initramfs in order to fix any corrupt file systems during boot. However, after jetpack 4.6 was introduced the approach I was using to build this image wasn’t booting correctly. I’m happier with the approach that make’s no change to the delivered initramfs now. Also, there were minor differences I found with the Xavier AGX. The approach I finally chose works with all of the nano, NX and Xavier AGX.

Cheers,
Kim

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.