SD card damage protection

Here is the image. GZipped. 6GB. User “nvidia”, password “nvidia”. System English, keyboard German.

Nothing special

https://www.dropbox.com/s/u04b8ihupyzetom/webrtc.rdb?dl=1

@linuxdev Could you please tell me, when I can remove the image from my dropbox? I’m missing space…

I see this now. If this were still running the scan process (for malware), then it might not have been there for me.

I downloaded the original “extlinux.conf” and “initrd” from the first post with those downloads, but the initrd.img-4.9.201-tegra (14.6 MB) download had expired. I used one of your newer links, but it only had the “initrd” and not the “initrd.img-4.9.201-tegra”. The final new link to dorpbox, which is here, has an error 404:
https://www.dropbox.com/s/u04b8ihupyzetom/webrtc.rdb?dl=1
…and I am guessing this is the initrd.img-4.9.201-tegra.

I am sorry the temp file upload systems are being a pain, but this is the one file I still need.

Sorry for all the hassle…

Yepp. I agree on 404, this is strange. I have created a new link

https://www.dropbox.com/s/ka25e9ymik8pizn/image.img.gz?dl=1

The file is the complete image I flashed onto the SD, gzipped. At least the download started here.

initrd.img-4.9.201-tegra (14.6 MB)

Attached the initrd.img-4.9.201-tegra again here.

Let me know.

The direct forum download link for “initrd.img-4.9.201-tegra (14.6 MB)” is not working.

@TomK is this a result of the file type and size?

I am downloading the dropbox “image.img.gz” now (download is not complete), but can you tell me if this is the initrd.img-4.9.201-tegra"?

EDIT: The image.img.gz download comleted. It is the initrd-4.9.201-tegra I am most in need of. If image.img.gz is a kernel, then I still need this, but if image.img.gz is initrd-4.9.201-tegra, then I have it. Still running gunzip so I don’t know what the file type is yet.

Yes, I agree. This is a PITA. I will upload it to my dropbox, once you have finished the download of the image (can delete it then).

Let me know please.

No, it is the cloned image after installation as I already told. Username “nvidia”, password “nvidia”, keyboard German, System english. You should be able to flash the gunzipped image if you like.

If not, disregard. I will delete it and uplaod the tegra

Got it. Gunzip will take a long time for the clone. Is this based on R32.5, or R32.5.1? Actually, after it completes gunzip, I should be able to see that myself with loopback mount. If I can’t get the initrd for 4.9.201-tegra, then I should be able to pull it out of the clone.

It is R32.5, the latest.

Will remove it and upload the tegra image. Stay tuned :)

I just realized, I need to verify something about the clone: This is for the SD card model on a dev kit carrier board? Or something else?

Here it is (hopefully)

https://www.dropbox.com/s/pjd3svp2vz6f2vv/initrd.img-4.9.201-tegra?dl=1

Let me know

Oh, wait. Let me doublecheck.

I just realized, I need to verify something about the clone: This is for the SD card model on a dev kit carrier board? Or something else?

It has been cloned from a 32 GB SD card. And was already used as such.

This will take a bit of time to look at, but I should have the start at this point. I’ll have to set up a Nano for this…not sure if my early Nano is a drop in replacement or not, but I’ll find out.

This will take a bit of time to look at, but I should have the start at this point. I’ll have to set up a Nano for this…not sure if my early Nano is a drop in replacement or not, but I’ll find out.

I’m having that B01 (?) model, the one with the two CSI slots.

Take your time, I’m thankful for your help. Did you download the tegra-image from the dropbox?

Regarding 32.5 vs. 32.5.1: Seems to be 32.5

 uname -r 
 4.9.201-tegra 
 head -n 1 /etc/nv_tegra_release 
 # R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t210ref, EABI: aarch64, DATE: Fri Jan 15 22:55:35 UTC 2021

Ping me in a few days if you don’t see a reply yet, but I am using the a02 model with a single CSI slot. I won’t be able to use the clone directly, but I will be able to look at the initrd (and the clone might still be useful). My NX has two CSI slots, but it uses CBoot only without U-Boot, so I can’t really substitute that either.

Perfect. Thanks!

Hi @linuxdev,

Unfortunately, the maximum attachment size is 10MB for the forums.

Tom

1 Like

I am not done looking at this, but wanted to post some notes so they are not lost. Regardless of succeeding or failing with OverlayFS I want to document some steps people might find useful in examining a clone of an entire disk (as opposed to a single partition), along with the initrd examination. Also, it isn’t much fun if teaching isn’t allowed, so part of the goal is to simply explain the theme behind what is going on rather than naming steps which don’t explain what is happening.

Some trivia you might find useful to learn, but is not really needed (skip this paragraph if you want): Keyboards and mice and a number of devices output a stream of bytes, and are known as “character” devices. Hard drives, SD cards, and most storage devices arrange data in “blocks”, and thus are known as “block devices”. In particular, a block device can be partitioned, and any software trying to read this has to understand the partitioning and formatting of the block device. In a way there is no difference between a file saved on disk, versus an actual hard drive the SATA controller might see. Any character device or block device will be accessed by a pseudo file in “/dev”. Looking at those files with “ls -l /dev/...something...” will show a leading “b” for block devices, or a leading “c” for character devices. “Covering” a file with “losetup” implies creating a “loopback” device special file of type “b” (block), and that block device causes the file to pretend it is a hard drive or a partition of the hard drive. We’re going extract content from the clone by pretending it is a block device.

Your clone image, “image.img”, can be examined as a disk via:

sudo losetup --find --show ./image.img
# Assuming it is "`/dev/loop0`"...
sudo gdisk -l /dev/loop0
# Excerpt of output:
# Sector size (logical/physical): 512/512 bytes
# ...
# Number  Start (sector)    End (sector)  Size       Code  Name
#    1           28672        62332927   29.7 GiB    8300  APP

# The rootfs starts sector 28672, and with 512 bytes per sector, the offset is 14680064 bytes.
# Detach loop0:
sudo losetup -D
# Attach again, but at offset...will assume it uses "/dev/loop0" again:
sudo losetup --show --find -o 14680064 ./image.img
# Mount the partition within the disk image to "/mnt":
sudo mount /dev/loop0 /mnt
# Now we can use the initrd and extlinux.conf files which are actually on your system.

Some files of interest:

# ls -l /mnt/boot/*initrd* /mnt/boot/*Image* /mnt/boot/extlinux/extlinux.conf
-rw-r--r-- 1 root root      845 Jan 19 14:12 /mnt/boot/extlinux/extlinux.conf
-rw-r--r-- 1 root root 34338824 Jan 19 13:59 /mnt/boot/Image
-rw-r--r-- 1 root root  7159844 Jan 19 13:59 /mnt/boot/initrd
lrwxrwxrwx 1 root root       24 Mar 15 10:58 /mnt/boot/initrd.img -> initrd.img-4.9.201-tegra
-rw-r--r-- 1 root root 15281358 Mar 15 10:59 /mnt/boot/initrd.img-4.9.201-tegra

I am assuming your “/boot/Image” is from the custom install. Can you confirm this is not the “stock” kernel? If it was, then it was probably added on Jan. 19. I am most interested in the initrd, so I’ve copied “/mnt/boot/initrd.img-4.9.201-tegra” to my local disk (preserving the loopback mount and not making changes to it). I also copied the “/mnt/boot/extlinux/*” files.

You could now:

# Make sure you first cd out of the mount point.
sudo umount /mnt
sudo losetup -D

Unpacking the initrd as a cpio archive:

# Create and cd to a temp location which is an empty directory, and if the actual initrd
# file is not there, either copy it in or name the full path to the alternate location. It is best
# to have the initrd image in another location since the unpack will not unpack to any subdirectory...
# when unpacking an initrd like this it unpacks to where your current directory actually is. If your
# initrd cpio file is there, then this will be in addition to the unpacked content. This is ok, except that
# if you make changes and plan on repacking this to become a new initrd, then you'll have to
# remember to move the old initrd cpio file somewhere else...you don't want an initrd with another 
# initrd inside it (there are size limits to an initrd).
sudo -s
gunzip < initrd.img-4.9.201-tegra > initrd-4.9.201-tegra.cpio
cpio -vid < initrd-4.9.201-tegra.cpio

There will now be a new “miniature” root filesystem in the directory you are in. What follows is from the main rootfs, and not from the “miniature” initrd partition unless specified.

The extlinux.conf names the kernel, so the kernel is from the relevant file and not from a partition (unless you’ve burned a fuse…make sure to confirm that the fuse is not burned and you are not using any secure boot). The extlinux.conf shows the initrd is the file “/boot/initrd”, which happens to be a symbolic link to:
initrd-4.9.201-tegra.cpio
“…this is the initrd you want…”

I see the “quiet” token is set in the kernel command line. For any testing you will want quiet removed for better serial console boot logging.

Now moving to the unpacked initrd-4.9.201-tegra “miniature” filesystem which was found inside of the clone.

The initrd will have a single file, “init”, and the rest will be subdirectories (the full filesystem will also have an init, though it will probably be in “/sbin” instead of “/”). When the Linux kernel starts it has basically only one purpose beyond its own initialization. Initialization is considered “kernel space”, and has process ID (PID) “0”. The application it starts outside of kernel space, “init”, is “user space”, and has PID 1. The “init” file is the beginning of everything you might be interested in. It is the bootloader (before Linux ever loads) which loads a kernel. The bootloader is a kernel, it just isn’t Linux, and the first thing the bootloader wants to run (after arranging content in memory) is a kernel. A bootloader ends its life by overwriting itself with the Linux kernel. The Linux kernel does not end its life when running init.

If the kernel loaded is the one in “/boot”. If that filesystem type is not readable, then boot fails. Similar for the initrd: It is in “/boot”, and the ext4 filesystem must be readable. The “/boot” content is often on its own partition so that it can always remain type ext4 and the bootloader can put content into memory before transferring execution to the Linux kernel.

Then comes the reason for an initrd, and probably the reason for OverlayFS not changing to read-only on the ext4 partition: If the Linux kernel does not have the functionality to work with OverlayFS right from the start, then that content must be added in the form of a kernel module (and it is certain that the bootloader itself cannot read OverlayFS, so it is up to Linux to understand OverlayFS). However, kernel modules are always in the location “/lib/modules/${uname -r}/kernel/”. This is part of the rootfs. If that rootfs is in a filesystem type not understood by the kernel, then it cannot load the module needed to read the module. The new filesystem type will fail.

Normally, if Linux tries to read a filesystem type for rootfs (excluding “/boot”), the kernel will panic and the system dies on the spot. However, OverlayFS is unusual in the sense that it is a layer on top of the already understood ext4. Instead of panicking for lack of OverlayFS ability the Linux kernel will attempt to read the content using its existing driver, the one for ext4. Which is read-write. It is nice that there is no kernel panic, but it won’t get your read-only OverlayFS on top of the ext4 hard drive partition.

The solution is to add a kind of “filesystem adapter”. The bootloader always understands what a cpio archive is (see the initrd unpack section…it’ll then make sense), and ext4. If the content is not on ext4, then the remaining choice is directly unpacking into RAM using the cpio initrd archive. The creator of the initrd has to make sure any Linux kernel modules needed for future rootfs content is inside of the initrd, and must use the init program of the the initrd (instead of the one in “/sbin” the real partition would use) to load kernel modules needed for loading the rootfs. The kernel modules in the real rootfs under the unknown filesystem cannot be used until those initrd modules are loaded. In theory, if the initrd has all required content for OverlayFS, and if the bootloader starts with the initrd instead of the “/sbin/init”, then one can use the odd filesystem rootfs after that. I think this step is missing in your initrd (I have not yet examined it).

The proper modules and content and other boot requirements are custom on Jetsons, and even though some OverlayFS install script might properly deal with an initrd on an older L4T release, it is also likely that changes to boot from one release to the next mean that installing OverlayFS to the initrd will be done incorrectly.

The place to start is to examine what kernel modules are present in the cpio archive (initrd after unpacked). Do the modules present include the OverlayFS content? To that extent you need to know what the “uname -r” command is. That command is nothing more than the kernel responding with what its version is, along with a custom name appended to the end of the version (which is set at the time of kernel compile). If your running system repsonds to “uname -r” with “4.9.201-tegra”, then you are in luck.

The reason for this being in luck is because:

  • The actual rootfs filesystem has directory “/lib/modules/4.9.201-tegra/kernel/”. Remember the clone loopback mount? Check this if it is still mounted:
    ls /mnt/lib/modules/4.9.201-tegra/kernel/
  • And lastly, the initrd itself must have that directory. If you unpacked the cpio initrd content into “~/unpack”, then you can verify:
    ls ~/unpack/lib/modules/4.9.201-tegra/kernel

If you ever compile a kernel and it has a different uname -r, then you will have lost all access to the modules (you’d have to install all modules at the new uname -r location). I think you are good to go on this, but you should verify after boot that “uname -r” responds with “4.9.201-tegra”. This means finding kernel modules can succeed.

Next, from the module location (cd /lib/modules/$(uname -r)/kernel/), do you have the required modules? The modules should be present in the correct subdirectory of this in both the cpio archive initrd and in the rootfs location (which is not part of the initrd).

I did not research required modules for OverlayFS, but here is an interesting file search (keep in mind I am assuming you have your entire rootfs loopback mounted on the host PC at “/mnt”, and that your unpacked cpio archive initrd is at “~/unpack”):

find ~/unpack/lib/modules/4.9.201-tegra/kernel -iname "*overlay*"
find /mnt/lib/modules/4.9.201-tegra/kernel -iname "*overlay*"

I find these files (including subdirectory name within the “kernel/” subdirectory) on the main rootfs, but entirely missing in the initrd:

fs/overlayfs
fs/overlayfs/overlay.ko

Thus, your initrd cannot act as an adapter to OverlayFS…the kernel function is missing. You will need to manually add those to an unpacked initrd, and then repack the initrd (there might be more steps, but this is a guaranteed minimum requirement if the kernel Image does not have that OverlayFS content integrated), followed by putting the new initrd back in the “/boot” location (preferably with a new name and the extlinux.conf edited to add a new kernel entry which matches in every way with the old entry, other than pointing at the new initrd name…then the old one is still present for backup).

This is a really long reply, so I will give information on unpacking and repacking an initrd in the next message of the thread. I will assume you’ve already read this one and gone through the steps to unpack the initrd. I’ll only show basics when I do this, am running short on time today.

Continuing the last message with more initrd information…

Assuming you unpacked your initrd into your host PC’s location “~/unpack”. Use “sudo” for all steps which follow unless otherwise mentioned.

When you build a kernel it has a subdirectory structure to keep down the clutter. The location of the subdirectory “fs/” is where filesystem drivers are arranged. The driver for “overlayfs” is itself in subdirectory “/fs/overlayfs/” (the names of files and directories are usually lower case…the exception is for script or configuration, e.g., the Kconfig files start upper case). If you were to build modules in the kernel, then you’d find the resulting “overlay.ko” kernel module at subdirectory “fs/overlayfs/overlay.ko”. *This subdirectory name will be a match in both the kernel source and in “/lib/modules/$(uname -r)/kernel/”. For example, in the full rootfs (not the initrd), the “find” command found “/lib/modules/4.9.201-tegra/kernel/fs/overlayfs/overlayfs.ko”. Your rootfs is complete, but you must duplicate this in the initrd.

So if you unpacked the initrd (remember to use sudo for what follows), then “cd ~/unpack/lib/modules/4.9.201-tegra/kernel/fs/” (if you didn’t unpack at “~/unpack”, then adjust). Then add the required directory:

mkdir overlayfs
cd overlayfs

Now assuming your actual rootfs (not the initrd cpio archive you are now at) is still mounted on “/mnt”:

cp /mnt/lib/modules/4.9.201-tegra/kernel/fs/overlayfs/* .

…you should now have the “overlay.ko” copied into the unpacked initrd’s correct module subdirectory. What you need to do now is to repack the initrd, give it a slightly different name, and copy it to the “/boot” of your actual rootfs (note that initrd content does not include any “boot/” subdirectory). Then add an entry to extlinux.conf to boot with most everything being the same, except it points at the new initrd and not the old one. When booting this OverlayFS might work. The kernel will support it prior to running a “pivot_root” to switch from the miniature initrd filesystem to the real filesystem.

The reverse of unpacking an initrd…the act of packing one…goes like this, and should be done with sudo (if you run “sudo -s”, then you remain root for all following commands until after you “exit”):

sudo -s
# root has a different home directory, make sure you use the home of your regular user.
cd /home/YourUserName
cd ./unpack
# create a cpio version in the parent directory...which should be "/home/YourUserName":
find . -print | cpio -ov > ../initrd-4.9.201-tegra.cpio
# You will now have "initrd-4.9.201-tegra.cpio" in the parent directory, so:
cd ..
# The cpio archive is compressed, and it will be ready after a rename:
gzip initrd-4.9.201-tegra.cpio
mv initrd-4.9.201-tegra.cpio initrd-4.9.201-tegra
# You now have the final intrd, file name "initrd-4.9.201-tegra"

Your original “/mnt/boot/extlinux/extlinux.conf” contains this boot entry:

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 

Let’s create a duplicate of this, and edit the duplicate slightly. So copy and paste the above with an empty line separating the two entries, and turn the new entry into this (removes “quiet” and changes the name of “initrd” to “initrd-4.9.201-tegra”, plus changes the identifying LABEL plus MENU):

LABEL overlayfs
      MENU LABEL overlayfs
      LINUX /boot/Image
      INITRD /boot/initrd-4.9.201-tegra
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 

If you know how to use serial console to interrupt boot at the kernel selection stage (let me know if you want to do this), then you would see two entries to pick from. Something like:

1. primary kernel
2. overlayfs

…you’d pick “2” and boot. You might still need other changes, e.g., to kernel command line, but it should now be possible to succeed. If normal OverlayFS docs (which I have not read) mention other changes, e.g., the “rw” kernel command line changing to “ro”, then we can work on that. Whatever change the standard docs might mention could also require a change in the initrd (especially it might need editing of the initrd’s “init” program…the program which the kernel itself runs first…it might have disk mounting operations in need of edit).

If you want to simply change the boot to default to the overlayfs, and don’t want to pick with serial console, then at the top of the extlinux.conf change “DEFAULT primary” to “DEFAULT overlayfs”. If something goes wrong with this, then you could use your serial console to pick the original “primary kernel” entry.

Very important to avoid frustration: There are some packages which can update through the normal apt mechanism. If a package changes the kernel (including modules), the initrd, or the extlinux.conf, then it might overwrite your changes. If you get things working and want to explore what packages are unsafe to update, then just add a post.