SD card damage protection

Consider me to be the last Linux dumbass, just capable of opening a terminal and type in some things.

OK, I flashed a R32 and this is the serial log for an unchanged start. Will now apply the scripts for the overlayfs and post the serial log then.

out.txt (22.6 KB)

Good. Applied the “lehni” scripts. System rebooted (boot log attached). Touched a file. Rebooted. Found the file again. Nice ReadOnly system, indeed…:(

T H I S D O E S N O T W O R K

out.txt (80.9 KB)

OK, I’ll leave Nvidia alone with their arrogant attitude to do things for specialist only. I guess the last company who did that successfully was Nokia. Going with RPI instead.

Thanks again.

Good bye

I think a better way to keep of the idiots is to make the box $ 3000. And to not wilder in the price segment of an RPI4 plus addons.

End of message :) Thanks anyway. Not your fault.

If someone of the knowing finally will come up with a receipt I consider to return to the arrogant

@dusty_nv Thanks for editing. I would like to sink the entire thread, if I could. It’s a painful demonstration of how powerless you can be when you have no idea and can’t find help or a solution.

For me it feels like “base camp 6” 100 m below the summit. I have to return, while my app is ready and works. I can’t protect the SD and I also will not go for $$$ for any eMMC board just for a POC. It is a real mess.

If anybody is following this: To change to mmcblk1p1 is not the right choice. mmcblk0p1 is correct. Otherwise no boot

Even if you work on something else (like the RPi) it might be worthwhile to continue on this particular project as well, so will try to answer more questions. I’m not sitting there to experiment, so it is more difficult to know what to do next without some possibly (probably?) confusing explanations.

In your file “/boot/extlinux/extlinux.conf”, do you see the word “quiet” as part of the “APPEND” line? If so, then remove the word “quiet”. Any editor will work. To see what is in the file without opening an editor you can “cat /boot/extlinux/extlinux.conf”. When editing your editor must be started as user root, which is done by prefixing “sudo” to the editor start command.

Regarding this:

…does this mean that after modifying that file you rebooted again and the change persisted? Changes should revert and go away after a reboot. However, you said “T H I S D O E S N O T W O R K”, so it sounds like the changes were permanent.

The SD card model uses “/dev/mmcblk0” since it doesn’t have eMMC and enumeration starts at 0. If this were an eMMC model, then the SD would be bumped to “/dev/mmcblk1”. It can be hard to always name the correct one when replying to many posts where people might have either model of Nano, so beware the “1” or “0” might be off in posts speaking of a different model. Summary:

  • For SD card models use “/dev/mmcblk0”,
  • for eMMC models the SD card is instead “/dev/mmcblk1”.*.
  • The first partition appends “p1” to the mmcblk#.

@linuxdev

Even if you work on something else (like the RPi) it might be worthwhile to continue on this particular project as well, so will try to answer more questions. I’m not sitting there to experiment, so it is more difficult to know what to do next without some possibly (probably?) confusing explanations.

I really admire you and appreciate your patience. I will try to resume, because I really think, it is worth it. It is just so close and I can’t find the reason for that.

In your file “ /boot/extlinux/extlinux.conf ”, do you see the word “ quiet ” as part of the “ APPEND ” line? If so, then remove the word “ quiet ”.

I admit I forgot to do that before. But I know how to do that and I will do next time

…does this mean that after modifying that file you rebooted again and the change persisted? Changes should revert and go away after a reboot. However, you said “T H I S D O E S N O T W O R K”, so it sounds like the changes were permanent.

Well, I did “touch test” and checked, that the file was there. Then I rebooted and I expected to see the file no longer, but it still was there. This in turn did show me, that the file must reside on my SD and not - as expected - in RAM.

  • For SD card models use “ /dev/mmcblk0 ”,

Yepp. Confirmed.

OK, how to proceed? I would suggest that I do these steps in a row:

  1. Flash a new image from the Nvidia site, basically the latest available. Bring that up, configure it, have the console logger aside

  2. Once this is up, edit /etc/boot/extlinux/extlinux.conf, copy the current state (for documentation) and remove “quiet”.

  3. Reboot and completely log the boot process via serial

  4. Apply the “lehni” repo. This basically does exactly, what is required to create a overlayfs: It creates a new “initramfs” and enters this as boot image. I would before boot check, if “quiet” removed is still the case (I would expect this to be the case).

  5. Then reboot and log the boot process. In case the system comes back I would make the “touch test” check again.

I could provide the boot log for both cases.

Would that be a good plan to follow?

Thanks again. You are pure motivation.

Yes. I also recommend just run updates to get them out of the way before doing more. If an update messes up something it is a lot of work for nothing. If updates are all in and the system is working normally, then you can clone the SD card and avoid all that work by restoring from the clone instead of doing all those steps again. From the host PC, be sure you have a lot of spare disk space (e.g., via “df -H”), and assuming on the host PC the SD shows as “/dev/sdb” (that’s just an example, it could be something else…if you monitor “dmesg --follow” during SD card plugin it will confirm what it actually is):

  1. Make sure the SD card is not mounted, e.g.,
    sudo umount /dev/sdb
  2. Use dd to clone:
    `sudo dd if=/dev/sdb of=sdcard_clone.img bs=1M
    (note that “bs=1M” has no effect other than using more buffer to help speed it up)
  3. After the clone is done, then you could restore to the same SD card after it is messed up and skip all those previous install steps and be ready (assumes SD card is not mounted, and that you first cd to the directory with the clone):
    sudo if=./sdcard_clone.img of=/dev/sdb bs=1M; sync

As long as the QSPI boot memory is not altered this will have you fully back up to an updated system with account setup and other settings without any more work. Then whatever you test can fail and you won’t have so much to worry about. If you reach a milestone and something gets further along, then you can clone again, and any restore will save that progress.

Rebooting and saving a serial console log is always a good idea if you might need to post something.

I don’t know about the “lehni” repo since I have not used it. I suspect that if it has a weakness, then it is from a need for different content in the initrd (initramfs) due to either the Jetson model’s software, or some change in the L4T release version (there is different boot content depending on L4T release even when the hardware is the same).

Incidentally, one of the biggest reasons for a boot log is to see what the initrd did. In theory, if the initrd echos what it does as it runs, then we’ll see information about how modules were loaded there, and the result. This is a good plan.

Yes, thanks for the clone hint. I have done this a lot of times with my RPI SDs.

“lehni-repo” is this GitHub - lehni/root-ro: Read-only root filesystem for Raspbian Stretch and Jetson Nano (using OverlayFS). If you check what is done in install-nano.sh, then it is simply doing initramfs and adding this as INITRD to the boot config. And create some mount scripts. So basically it is what all the world is saying what has to be done to create an overlayfs.

BTW: After the installation there is also already such an image in the same directory the /etc/boot/extlinux/extlinux.conf. You can achieve the same endless bootloop by adding this as INITRD

OK, I will start to flash the initial image.

OK, groundhog day again.

  1. Flashed canned image. Changed “quiet”, rebooted.

Bootlog:
first.txt (80.3 KB)

  1. Applied “lehni”

    neil@jetson:~/root-ro$ sudo ./install-nano.sh
    dphys-swapfile is not installed, assuming we dont need to disable swap
    Setting up maintenance scripts in /root…
    Setting up initramfs-tools scripts…
    Adding “overlay” to /etc/initramfs-tools/modules
    Updating initramfs…
    update-initramfs: Generating /boot/initrd.img-4.9.201-tegra
    Warning: couldn’t identify filesystem type for fsck hook, ignoring.
    I: The initramfs will attempt to resume from /dev/zram3
    I: (UUID=5753da45-79ca-4e0a-877c-5ccdbaa84457)
    I: Set the RESUME variable to override this.
    /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_EGL.conf: No such file or directory
    /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_GL.conf: No such file or directory
    Changing INITRD in /boot/extlinux/extlinux.conf
    Removing the random seed file
    Please restart your Jetson Nano now to boot into read-only mode
    neil@jetson:~/root-ro$

  2. Attempt to setup “permanent RO” failed:

    neil@jetson:~/root-ro$ sudo /root/reboot-ro
    The root filesystem is already in readonly mode.

  3. Rebooted then normally:

    neil@jetson:~/root-ro$ sudo reboot
    Connection to 192.168.188.87 closed by remote host.
    Connection to 192.168.188.87 closed.

Bootlog:
second.txt (96.6 KB)

Result: Endless boot…

Re-flashing now.

There must be a general problem with this approach in later versions of JP. With an R32 image at least this endless boot loop does not appear, but the system after the patch wasn’t really read-only. (THISDOESNOTWORK comment). I now entered the contained INITRD.IMG, which I found after the fresh installation, and the same symptom of an endless boot loop appears.

Say goodbye to overlayfs on later versions?

Found image:

neil@jetson:~$ ls -lall /boot/initrd.img
lrwxrwxrwx 1 root root 24 Mär 13 22:10 /boot/initrd.img -> initrd.img-4.9.201-tegra

FYI, one of the biggest “moving targets” of L4T releases is boot code. Everything leading up to and including the kernel load changes rapidly, while the actual Linux o/s does not change much. Anything touching the boot code such that it makes assumptions from another L4T release is probably going to fail. My R32.1 kernel is “4.9.140-tegra”, while I see the script you used is naming “4.9.201-tegra”. My R32.4.3 release also shows kernel “4.9.140-tegra” (there is an initrd involved in R32.4.3 as a default, so this is an example of changing boot requirements depending on L4T release). I don’t currently have an R32.5.x release installed, but what do you see from these commands?

uname -r
head -n 1 /etc/nv_tegra_release
ls -l /boot/Image*
ls -l /boot/initrd*

(I’m thinking there may be a kernel version change used in support of OverlayFS, and perhaps the kernel release and build may be at least at fault for whatever L4T release you are using)

Also, I’m not sure if the forums will upload some file types, but could you attach your “/boot/extlinux/extlinux.conf”? You might need to rename it “extlinux.txt”. I’d also like a copy of any “/boot/initrd*”. Beware that when you run “ls -l /boot/initrd*” a symbolic link will possibly show up with an arrow “initrd.img->...” pointing at some other file, and I do need to know about the sym link, but I only need a copy of the actual initrd file it points at. On my R32.4.1 system I see:

root@nx:/boot# cd
root@nx:~# ls -l /boot/initrd*
-rw-r--r-- 1 root root  5565615 Jun 25  2020 /boot/initrd
lrwxrwxrwx 1 root root       24 Jun 23  2020 /boot/initrd.img -> initrd.img-4.9.140-tegra
-rw-r--r-- 1 root root 15231643 Jan 28 13:51 /boot/initrd.img-4.9.140-tegra

…and in this case I’d be interested in a copy of “/boot/initrd” and “/boot/initrd-4.9.140-tegra”. I am only guessing, but your script may have installed another initrd (probably it would have “4.9.201-tegra” in its name, but that is only a guess).

You could make these files smaller by copying them somewhere else, and then running “bzip2 -9 initrd...” (use the actual file name), and this would compress to become “initrd...something.bz2” (the original file would be renamed the same thing with a “.bz2” file name extension). If you cannot attach these to the forum, then we can find another way to get the files.

Once again, I am only guessing, but I suspect that the issue is somewhere within the combination of kernel file, initrd files, and firmware related to those. All of these are related to boot.

One clue, which I don’t really know if it is involved yet, is that on your boot log I see:

## Flattened Device Tree blob at 83100000
   Booting using the fdt blob at 0x83100000
ERROR: reserving fdt memory region failed (addr=0 size=0)
ERROR: reserving fdt memory region failed (addr=0 size=0)

…perhaps a kernel change needed to work with those memory regions is incompatible, and if the init software is mixing up 4.9.140-tegra and 4.9.201-tegra, perhaps it is as simple as getting a consistent kernel configuration both in and out of the initrd. I can unpack any initrd and figure out the flow of boot and hopefully make a guess at anything broken due to different kernel releases and device tree.

That sounds pretty elaborated and I will be happy to provide you with all the required info. But out of curiosity: Wouldn’t it be way easier if you would simply follow the two lines below to set it all up on your device? Honestly I would pay you for your efforts. Just PM me and we agree on something?

Steps

  • download and flash latest Nvidia provided JP4.5 image
  • apply the „lehni-scripts“

Again I would compensate you for that if I in the end would have a receipt for how to mske my SD RO

But I will also provide the info later

hello foreverneilyoung,

please check this, if your goal by making sd-card as readonly for production systems.
Can Jetson developer kits be used as production systems or as part of a product?

Jetson Nano Developer Kits (i.e. SD-card versions) are NOT for production use.
The Developer Kit is used to develop and test software in a pre-production environment.


it’s my curiosity why you claim lehni works for Jetson Nano?
it did not show any functional JetPack release version in the repository description.
besides, I would also doubt it works with JetPack releases since its tested image was used-on kernel-4.14, whereas the latest JetPack-4.5 is kernel-4.9.


for your reference,
we had similar discussion thread, Topic 60860 to make rootfs as read-only using overlayfsroot. this were test by earlier JetPack release versions. (i.e. JetPack-3.2).
here’s also a discussion thread, linuxquestions.org, which talking about how to setup a readonly rootfs without an initramfs.
thanks

Jetson Nano Developer Kits (i.e. SD-card versions) are NOT for production use.
The Developer Kit is used to develop and test software in a pre-production environment.

And what? I could now say, I don’t intend to use it in production, just for a POC, which would be true. But who cares? I lately read such a note on an ordinary bucket: “Please note this is just for wish water. Don’t use it as toilet. Use public rest rooms instead.” You know what? I ignored it and the bucket didn’t melt away…

I don’t want to kill the king, I just want to do something, which is in other Linux systems obviously the easiest thing to achieve. Even the “lehni” thing: It is based on a script written for a very old RPI version, and it is still running on the latest Raspbian OS.

This argument is ridiculous, to say it politely. You should rather help me to make it work, if you drop in, because that would be the way to go, not to say: DONT’ DO THAT, IT’S THE FORBIDDEN PALACE.

it’s my curiosity why you claim lehni works for Jetson Nano?

Not my claim. The claim of the owner of the repo. I had a thread with him, in which I showed him, that it doesn’t work. overlayfs for Jetson - #19 by juerg
He said that it worked for him on an earlier kernel. I tried that too. It didn’t work for me either on older versions, it didn’t go into an endless boot, but it wasn’t read only either…

for your reference,
we had similar discussion thread, Topic 60860 to make rootfs as read-only using overlayfsroot. this were test by earlier JetPack release versions. (i.e. JetPack-3.2).

Yes I know these, and as far as I can see, none of these did end up in something reproducible today. Or do you see this? What do you want to tell me with this argument? I would be happy if you could show me a working solution for the simple question “HOW TO MAKE MY SD RO AND THE ENTIRE STUFF IS STILL WORKING”, because with my RPI background I would expect this should be possible, otherwise please don’t stop me here

Thanks

I don’t currently have an R32.5.x release installed, but what do you see from these commands?

Here you are:

neil@jetson:~$ uname -r
4.9.201-tegra
neil@jetson:~$ head -n 1 /etc/nv_tegra_release
# R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t210ref, EABI: aarch64, DATE: Fri Jan 15 22:55:35 UTC 2021
neil@jetson:~$ ls -l /boot/Image*
-rw-r--r-- 1 root root 34338824 Jan 19 21:59 /boot/Image
neil@jetson:~$ ls -l /boot/initrd*
-rw-r--r-- 1 root root 7159844 Jan 19 21:59 /boot/initrd
lrwxrwxrwx 1 root root 24 Mär 14 16:20 /boot/initrd.img -> initrd.img-4.9.201-tegra
-rw-r--r-- 1 root root 15282367 Mär 14 16:21 /boot/initrd.img-4.9.201-tegra

Also, I’m not sure if the forums will upload some file types, but could you attach your “ /boot/extlinux/extlinux.conf

Here is the file:

extlinux.conf (845 Bytes)

initrd (6.8 MB)

initrd.img-4.9.201-tegra (14.6 MB)

Please note: All these files are in the state after a fresh installation of JP4.5. No sudo apt update && sudo apt upgrade. Just flash and configure. No “lehni-script” applied yet.

If I change /boot/extlinux/extlinux.conf that way, that

INITRD /boot/initrd

is

INITRD /boot/initrd.img

the system does not return. It goes into an endless boot. This is the symptom I’m seeing now since several weeks, whenever I try to make that overlayFS run.

Thanks for your time. Again, I would compensate you for that, I don’t expect you to help me without. My mail is my forum name plus ‘at googlemail.com

@linuxdev: Since you have asked for it on another thread (overlayfs for Jetson - #29 by hegnozespu):

Here is mine:

neil@jetson:~ cat /proc/device-tree/chosen/bootargs tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 gpt tegra_fbmem=0x800000@0x92ca9000 is_hdmi_initialised=1 earlycon=uart8250,mmio32,0x70006000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 neil@jetson:~

I would end up backing up (via clone) the current installation, and it is possible I could work on this locally and find an answer. However, there is no guarantee that anything I find would be the same as on your end. The local environment can itself have major effects. Even something as simple as character set and language on the host o/s can drastically change what happens. I’d really like to see the initrd and device tree actually running for your case. It sounds complicated, but they are just files…the only issue is if you can directly attach them to the forum thread, and if not, find an alternate file upload site. There is no guarantee that doing this locally would be fast, or even succeed at all, so I’d rather start with examining those files (and if I did try local install, then I’d still want to compare a working install against your initrd and device tree).

OK, will do a clone and provide it to you somehow :)

It is pretty much impossible to detect and access the SD after having plugged it into my Ubuntu. I have done this clone several times now with RPI images, I have scripts, which shrink the PI image and finally gzip it.

But here something strange happens: The device is there, it is /dev/sdc. Ubunutu reacts on plug/unplug. But neither sudo unmount /dev/sdc works nor sudo dd....