My goal is to create an image based on OSTree that can be flashed to a Jetson Xavier. The script I use to create the rootfs is below. I’ve verified that I can boot the changes to the image, but I have to make them on an already-booted device. My issue is that the flash.sh script manipulates the rootfs before sending it to the device. Some of these changes might be required for for EFI boot, but I don’t know which steps. I’m currently going through the flash.sh script line-by-line to figure out what is needed to properly flash this rootfs, but I’m hoping there is a work-around or set of instructions that could save me from walking through every line. As an example, the files in /boot are moved to /usr/lib/modules/{kernel}/ and renamed (/boot/Image → /usr/lib/modules/4.9.140-tegra/vmlinuz, as an example). Does anyone have any advice on how to flash the rootfs to the device with minimal changes (and list those minimal changes for manual modifications)?
#!/bin/bash
set -euo pipefail # Exits script on any error.
# To run this script a few base packages are required. Run the following command
# to install this base packages:
#
# apt install -y build-essential debootstrap qemu-user-static ostree
#
# Requires file Jetson_Linux_R32.4.4_aarch64.tbz2 to be in the same directory as
# this setup.sh script.
HOMEDIR="/home/$USER"
echo "Will be using SSH key from the following home directory: $HOMEDIR"
# Start fresh and remove any existing builds. Existing root filesystems may have
# the immutable flag set, so the flag needs to be cleared before the directory
# can be removed.
sudo chattr -i out/Linux_for_Tegra/rootfs/ostree/deploy/mine/deploy/* || /bin/true
sudo rm -rf out || /bin/true
mkdir out
cd out
# Extract the Jetson tools
tar -xf ../Jetson_Linux_R32.4.4_aarch64.tbz2
cd Linux_for_Tegra
# Build the filesystem.
sudo qemu-debootstrap --arch=arm64 bionic rootfs
sudo /bin/bash -c "echo \"deb http://ports.ubuntu.com/ubuntu-ports bionic main universe\" > rootfs/etc/apt/sources.list"
sudo chroot rootfs /bin/bash -c "apt update && apt install -y libdrm-common libdrm2 libegl-mesa0 libegl1 libgbm1 libglapi-mesa libglvnd0 libwayland-client0 libwayland-server0 libx11-xcb1 libxau6 libxcb-dri2-0 libxcb-dri3-0 libxcb-present0 libxcb-sync1 libxcb-xfixes0 libxcb1 libxdmcp6 libxshmfence1 multiarch-support device-tree-compiler fontconfig fontconfig-config fonts-dejavu-core libcairo2 libdatrie1 libevdev2 libfontconfig1 libfreetype6 libgles2 libgraphite2-3 libgudev-1.0-0 libharfbuzz0b libinput-bin libinput10 libjpeg-turbo8 libmtdev1 libpango-1.0-0 libpangoft2-1.0-0 libpixman-1-0 libpng16-16 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libthai-data libthai0 libunwind8 libwacom-bin libwacom-common libwacom2 libwayland-cursor0 libwayland-egl1-mesa libx11-6 libx11-data libxcb-render0 libxcb-shm0 libxext6 libxkbcommon0 libxrender1 python python-minimal python2.7 python2.7-minimal gstreamer1.0-plugins-base iso-codes libasound2-data libcdparanoia0 libogg0 libopus0 liborc-0.4-0 libtheora0 libvisual-0.4-0 libvorbis0a libvorbisenc2 libgstreamer-plugins-bad1.0-0 libasound2 libpangocairo-1.0-0 openssh-server network-manager ostree && rm -rf /var/lib/apt/lists/*"
sudo mkdir -p rootfs/root/.ssh && sudo cp $HOMEDIR/.ssh/id_rsa.pub rootfs/root/.ssh/authorized_keys
sudo ./apply_binaries.sh
sudo rm -f rootfs/lib/systemd/system/nv-oem-config*.*
sudo rm -f rootfs/usr/sbin/nv-oem-config-firstboot
sudo rm -f rootfs/etc/systemd/nv-oem-config*.sh
sudo rm -f rootfs/etc/nv-oem-config.conf.*
sudo rm rootfs/README.txt
# Adapt filesystem to OSTree
# - /bin -> /usr/bin
sudo mv rootfs/bin/* rootfs/usr/bin/ && sudo rmdir rootfs/bin && sudo ln -s /usr/bin rootfs/bin
# - /sbin -> /usr/sbin
sudo mv rootfs/sbin/* rootfs/usr/sbin/ && sudo rmdir rootfs/sbin && sudo ln -s /usr/sbin rootfs/sbin
# - /home -> /var/home
sudo mkdir rootfs/var/home && sudo rmdir rootfs/home && sudo ln -s /var/home rootfs/home
# - /lib -> /usr/lib
sudo mv rootfs/lib/aarch64-linux-gnu/* rootfs/usr/lib/aarch64-linux-gnu/ && sudo rmdir rootfs/lib/aarch64-linux-gnu
sudo mv rootfs/lib/systemd/* rootfs/usr/lib/systemd/ && sudo rmdir rootfs/lib/systemd
sudo mv rootfs/lib/* rootfs/usr/lib/ && sudo rmdir rootfs/lib && sudo ln -s /usr/lib rootfs/lib
# - /opt -> /var/opt
sudo mv rootfs/opt/* rootfs/var/opt/ && sudo rmdir rootfs/opt && sudo ln -s /var/opt rootfs/opt
# - /root -> /var/roothome
sudo mv rootfs/root rootfs/var/roothome && sudo ln -s /var/roothome rootfs/root
# - /media -> /run/media
sudo rmdir rootfs/media && sudo ln -s /run/media rootfs/media
# - /etc -> /usr/etc
sudo mv rootfs/etc rootfs/usr/etc
# Setup OSTree
# sudo mkdir rootfs/sysroot
# sudo ln -s /sysroot/ostree rootfs/ostree
# Setup OSTree-Systemd tmpfiles.
sudo bash -c 'cat <<EOF > rootfs/usr/lib/tmpfiles.d/ostree.conf
L /var/home - - - - ../sysroot/home
d /var/roothome 0700 root root -
d /var/opt 0755 root root -
d /var/local 0755 root root -
d /run/media 0755 root root -
EOF'
# Move Linux, initrd and devicetree to /usr/lib/modules/${version}
modules="rootfs/usr/lib/modules/4.9.140-tegra"
sudo mv rootfs/boot/tegra194-p2888-0001-p2822-0000.dtb ${modules}/devicetree
sudo mv rootfs/boot/Image ${modules}/vmlinuz
sudo mv rootfs/boot/initrd ${modules}/initramfs.img
# Clear /dev so the OS can be commited.
sudo rm -rf rootfs/dev/*
# Create a repo and commit the OSTree image.
sudo mkdir repo
sudo ostree init --repo=repo --mode=archive
sudo ostree --repo=repo commit --branch=mine rootfs
# Create a fresh rootfs.
sudo mv rootfs orig_rootfs
sudo mkdir rootfs
# Create the OSTree image.
sudo ostree admin init-fs rootfs
sudo ostree --sysroot=rootfs admin os-init mine
sudo ostree --repo=rootfs/ostree/repo remote add mine http://10.0.1.4:8080 --no-gpg-verify
sudo ostree --repo=rootfs/ostree/repo pull-local --remote=mine repo mine
sudo ostree --sysroot=rootfs admin deploy --os=mine mine:mine
# Inform user of next step.
echo "To flash device run: sudo ./flash.sh jetson-agx-xavier-devkit mmcblk0p1"
What is the purpose to use that?
What kind of manipulation are you talking about?
We are not officially supporting the EFI boot.
If you want to move some files in rootfs, why not just operate them in rootfs folder? Didn’t you already chroot to rootfs and do that work?
OSTree, which OSes such as Fedora’s SilverBlue and Endless OS are built upon, is a tool used to create immutable Linux images that can be atomically upgraded in such a way that failures to upgrade do to intermediate upgrade states is impossible… either you boot on the old, pristine OS or on the new, pristine OS. All parts of the OS are upgraded at once. The components are stored on the device in a git-like repo, so files common between upgrades don’t take additional disk space. This is a common technology for IoT devices that need to handle their own upgrades in a safe way (no fear of corruption with ‘do not power off device while upgrade in progress’.
The changes that occur are those required by UsrMove, where all components of the OS are moved from /sbin, /bin, /etc (default /etc configuration… not configuration changes), /lib, /root and /home into locations either under /usr or, in the case of files that are not part of the immutable image (/home, /root, /opt), into the /var directory. I have made most of these changes successfully, already. I’ve even moved /boot/Image, /boot/initrd and the device tree into /usr/lib/modules/{kernel}/ successfully (though, because the flash.sh script inserts them into /boot during flashing, I have to move them on-device after first boot… one of the issues I would like to correct).
The EFI boot was an assumption and not a requirement. I’ve noticed that when I boot from the kernel image managed by OSTree (before flash.sh is executed), that the device fails to start. However, I can boot from and move to other locations the Image file inserted by flash.sh. Thankfully, flash.sh leaves /boot/extlinux/extlinux.conf alone, so my boot options are preserved, but I’m running into the following questions:
Why does the device not boot from the /boot/Image that exists after the apply_binaries.sh step? (This is the kernel I move and commit to OSTree to manage.)
What steps does flash.sh do to modify rootfs to make it bootable (minimal changes that need applied for the hardware to boot the device)?
If I make those changes myself, how can I get flash.sh to NOT change rootfs while flashing the rootfs to the device?
At the moment I’m going line-by-line through the flash.sh script to discover these changes and it is very tedious, especially not know exactly which steps are actually required to create a bootable image and which are not. At the same time, I’m going through the Nvidia boot documentation for the device in between device flashes. I’m hoping that I can get some answers to those questions that will accelerate my work so we can get our device to market, sooner.
The flash.sh is using tegraflash.py. Thus, you may also need to check the python script too.
Why does the device not boot from the /boot/Image that exists after the apply_binaries.sh step? (This is the kernel I move and commit to OSTree to manage.)
The device is booting from the /boot/Image. What causes you think it is not?
If I make those changes myself, how can I get flash.sh to NOT change rootfs while flashing the rootfs to the device?
There is a command “–no-flash” in flash.sh. Which will only generate binaries but not flash them.
I saw that bit about tegraflash.py. Just haven’t made it that far in the script.
After I flash the device and log in, I can move /boot/Image, /boot/initrd and the devicetree file into the /usr/lib/modules/{kernel}/ directory, then modify /boot/extlinux/extlinux.conf to point to those locations (according to the documentation, the only entry file that the boot process needs to discover is extlinux.conf… and that matches my experiment results). Having said that, if I move /boot/Image after apply_binaries.sh and before flash.sh, update extlinux.conf with the new paths, then flash the device, it will not boot. If I changes the paths to the /boot/Image that was inserted WHILE running flash.sh, it will boot. If I move (after flashing) the /boot/Image that was inserted WHILE flash.sh was running, it will continue to boot. If I move the /boot/Image file that was inserted BEFORE flash.sh back to /boot/Image, it will not boot. This tells me that flash.sh does something to /boot/Image or gets information about /boot/Image that it stores elsewhere that is required during flashing or booting… I just don’t know what it is or what it does.
As for the --no-flash… that is exactly opposite of what I want. --no-flash makes changes to the rootfs and doesn’t flash the device (and, of course, creates the image for flashing)… I want something that does NOT touch the rootfs and flashes the device ‘as is’.
I hope that helps clear up my findings and requirements. Thank you, again! (my next response may be next morning if you have any questions or suggestions for me to try)
I’m not sure, I’ll investigate that in particular in the morning. I’ve been just going top-to-bottom through the logic, so I haven’t made it to where that function has been called, yet. (Because I don’t know if the required changes are limited to the kernel, I’ve been exploring all changes.) I’ll jump straight to that function in the morning.
I’ve managed to work through most of the issues. The remaining issues don’t prevent me from moving on, but they are an annoyance. In the process of creating the filesystem, I end up relocating the Image, initrd and *.dtb files. The flash.sh script puts those files back on the filesystem. As a work around, I have an updater service that looks for and deletes those files on first boot, but it is still a bit of an annoyance as it makes the image larger and litters the filesystem with, essentially, useless files that will be immediately deleted on first boot. Because these files are added in early steps, is there no way to prevent flash.sh from inserting them right before pushing the image to the device?
So, I just updated to using Tegra186_Linux_R32.5.0_aarch64.tbz2 and the results I’m having are MUCH worse. It’s still the case where the flash.sh is trying to be too smart about what is in the rootfs directory (and it is actually risking the integrity of my own filesystem in the process). OSTree (as well as current trends in Linux OSes) moves everything to the /usr directory (with the exceptions of /etc/ and /var). You can see this in newer versions of Ubuntu and, more so, in Fedora Silverblue. To keep temporary backward compatibility, shortcuts are made on the rootfs (/lib → /usr/lib, /sbin → /usr/sbin, etc…). OSTree goes so far as bind-mounting /usr (which means that by the time I’m ready to deploy the OS, I don’t even have a /usr directory) to a set of hardlinks that make up the file system. As a temporary measure, I copied in a whole copy of /usr, but I hang on issues during flash.sh such as:
This specific command (due to the fact that (/lib → /usr/lib) is actually accessing MY HOST computer and not the rootfs (the links are correct after being loaded on the hardware). Basically, this is making a lot of (incorrect) assumptions about the layout of the file system (I make sure everything is correctly installed prior to running flash.sh). I had everything working on a 4.4 (though I created a service that deleted the files that flash.sh copied into the file system on first boot, so, again, very wasteful), but these changes are making it very difficult to do an initial deployment of an immutable OS with reliable, atomic updates. We want to use Nvidia, but my manager is starting to push us to try different platforms (reliable updates are more important than the hardware, for our company (exacerbated by being limited to 18.04 when most of our development is for 20.04)). Are there not any flags that could be added to prevent modifying the rootfs when flashing?
@halverneus Hi, Regarding the research about Ostree on Jetson Xavier, did you eventually succeed? Is there anything you can share about it? I am also planning to work with Ostree on Jetson.
I eventually gave up for another solution, unfortunately. Since then, I’ve also switched jobs, so I no longer work with the Jetson devices. A stepping stone I figured out was that I couldn’t directly flash the Nvidia device with an OSTree filesystem because flash.sh would attempt to modify the filesystem during the flashing operation, so I switched to flashing a base image, then using a USB drive with a desired image pre-loaded to write the OSTree filesystem. I ran out of time, but the intent was to let the flash.sh script prepare the image for flashing, then make UsrMerge/OSTree-specific changes to the resulting filesystem. That filesystem could be written to the harddrive via a USB installer without flash.sh making further changes. I did all these steps for my work, but I never was able to get back to using OSTree, sadly.