Ok, after not seeing my family and friends or doing anything else in my free time for 4 days in a row I think I’ve got it. Here are my notes in case it helps someone else. I’m specifically targeting Waveshare’s JetRacer AI PRO image but the idea should work the same for NVidia images:
We have two problems here: 1) U-boot doesn’t support F2FS, so the /boot directory where the kernel and extlinux config stuff lives must still be in an ext4 partition, but since it’s barely containing /boot it can be as short as 1GB and the rest of the contents of the old APP will go in a new partition (/dev/mmcblk0p15) named “ROOT” (for example). So we will destroy the APP partition and recreate it as two new partitions, a new 1GB ext4 APP partition with just /boot in and an F2FS partition with everything else. Problem number 2) is that the kernel itself doesn’t support F2FS either. For some reason NVidia thought that it would be funny to have Libreoffice but not F2FS support, so we need to recompile the kernel like we are in 1993. We are going to need Waveshare image which is based on Jetpack 4.5, L4T 32.5.0 and kernel 4.9.201 and I’m sticking to that exact version.
To compile the kernel you need ubuntu 18.04. Since I have 20.04, I used docker to pull a bare 18.04 image but you can skip this if you have 18.04 or use a 18.04 VM:
docker pull ubuntu:18.04
docker run --name jetracer4.9.201 -v /space:/space -it ubuntu:18.04
If you exit for some reason, you can come back in with:
docker start jetracer4.9.201
docker exec -it jetracer4.9.201 bash
Inside your 18.04 ubuntu, do:
useradd -G sudo -s /bin/bash -m javi
passwd javi # give it a password
apt update
apt install gnupg2 apt-utils xxd kmod sudo wget tzdata
apt install /space/JETRACER/sdkmanager_1.6.1-8175_amd64.deb
su - javi
sdkmanager --cli install --logintype devzone --product Jetson --host --targetos Linux --version 4.6 --target JETSON_NANO_TARGETS --flash skip
This will install sdkmanager, which I previously downloaded in /space/JETRACER
and note that I shared my host machine’s /space
dir as /space
inside the docker container.
Authenticate, accept all licenses, install all host stuff, if there is something that requires interactive configuracion CTRL-C, install it with apt and relaunch sdkmanager since sdkmanager won’t let you interact with dpkg.
Finally, skip installing SDK components on your Jetson Nano.
After this, we’ll be mostly following directions from this great blog: Compiling Custom Kernel Modules on the Jetson Nano | Kevin's Blog
wget http://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
tar xf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
export CROSS_COMPILE=$(pwd)/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export LOCALVERSION=-f2fs
wget https://developer.nvidia.com/embedded/L4T/r32_Release_v5.0/sources/T210/public_sources.tbz2
tar -xjf public_sources.tbz2
cd Linux_for_Tegra/source/public
tar -xjf kernel_src.tbz2
cd kernel/kernel-4.9
sudo apt install build-essential bc libncurses5 libncurses5-dev
TEGRA_KERNEL_OUT=$HOME/t4l-kernel
mkdir -p $TEGRA_KERNEL_OUT
make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
make ARCH=arm64 O=$TEGRA_KERNEL_OUT menuconfig
Go to File SYstems → enable F2FS filesystem support. Make sure you make it <*>
, not <M>
, ie. baked in support and not module support since module support would also require to modify initrd
and this is already tedious as it is.
To compile and pack the kernel + modules:
make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j32 # 32=num of threads
mkdir ~/to_tar
make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=~/to_tar/
mkdir -p ~/to_tar/boot
cp ~/t4l-kernel/arch/arm64/boot/Image ~/to_tar/boot/
cp ~/t4l-kernel/arch/arm64/boot/dts/* ~/to_tar/boot/
cd ~/to_tar
tar czf /space/JETRACER/kernel-4.9.201-f2fs.tgz .
This fixes problem NUMBER 2, ie. we now have a kernel that supports F2FS. Now to the partition reshuffle:
Download waveshare jetracer pro ai image from: JetRacer Pro AI Kit - Waveshare Wiki outside of the docker (I put it in a directory named /space/JETRACER):
unzip jetracer_pro_ws.zip
sudo losetup -Pf jetracer_pro_ws.img
mkdir APP ROOT OLDAPP
sudo mount /dev/loop0p1 APP
sudo mksquashfs APP jetracer_pro_ws.squashfs -comp lz4
sudo umount APP
sudo gdisk /dev/loop0
Follow this to destroy and recreate the partitions. I’m giving the root partition 30GB so that later on I don’t have to reflash the whole SD.
Command (? for help): d
Partition number (1-14): 1
Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-123596766, default = 28672) or {+-}size{KMGTP}:
Last sector (28672-123596766, default = 123596766) or {+-}size{KMGTP}: +1G
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): c
Partition number (1-14): 1
Enter name: APP
Command (? for help): n
Partition number (15-128, default 15):
First sector (34-123596766, default = 2125824) or {+-}size{KMGTP}:
Last sector (2125824-123596766, default = 123596766) or {+-}size{KMGTP}: +30G
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): c
Partition number (1-15): 15
Enter name: ROOT
Finally, exit with w
and confirm.
I’m backing up the original contents of APP into an squashfs because it keeps all permissions, owners, hard links etc and will be using rsync later on to split into the /boot and / partitions.
sudo partprobe /dev/loop0
sudo mkfs.ext4 /dev/loop0p1
sudo mkfs.f2fs /dev/loop0p15
sudo mount -o loop jetracer_pro_ws.squashfs OLDAPP
sudo mount /dev/loop0p1 APP
sudo mount /dev/loop0p15 ROOT
sudo rsync -aAHx --info=progress2 --delete OLDAPP/boot APP
sudo rsync -aAHx --info=progress2 --delete --exclude boot OLDAPP/ ROOT/
Replace kernel (save previous kernel and dtbs). In the end we’ll have /boot/4.9.201-tegra with the original kernel + dtbs and /boot/4.9.201-f2fs with our F2FS kernel, and by symlinking one or the other directory’s contents we can switch kernels.
cd APP/boot
sudo mkdir 4.9.201-tegra 4.9.201-f2fs
sudo mv Image tegra* 4.9.201-tegra
cd ..
sudo tar xf /space/JETRACER/kernel-4.9.201-f2fs.tgz ./boot
cd boot
sudo mv Image tegra* 4.9.201-f2fs
sudo ln -s 4.9.201-f2fs/* .
Now:
sudo vi extlinux/extlinux.conf
And replace:
APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
with:
APPEND ${cbootargs} quiet root=/dev/mmcblk0p15 rw rootwait rootfstype=f2fs console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
Unpack the modules in the new root fs:
cd ../../ROOT
sudo tar xvf /space/JETRACER/kernel-4.9.201-f2fs.tgz ./lib/modules
cd etc
sudo vi fstab
Replace:
/dev/root / ext4 defaults 0 1
with:
/dev/root / f2fs defaults,noatime 0 1
Then clean up:
cd ../../
sudo umount ROOT
sudo umount APP
sudo gdisk /dev/loop0 # p to print the partition table, it will be useful soon
sudo losetup -D
And finally flash the SD card (replace /dev/sdc with your SD)
sudo dd if=jetracer_pro_ws.img of=/dev/sdc count=31758 bs=1M status=progress
I limited to 31758 GB because that’s where the the new ROOT partition ends. To find the number I used gdisk /dev/loop0
earlier, printed the partition table with p
and found the last used sector of the ROOT (15th) partition plus 1, times 512 (sector size) and divided by (1024 * 1024) to find that 31758 number which is the used portion of the SD in GB and it doesn’t make sense to flash beyond it.
Put the card in the jetson, boot, rejoice:
jetson@jetson-desktop:~$ mount |grep f2fs
/dev/mmcblk0p15 on / type f2fs (rw,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,fsync_mode=posix)
TODO: script to auto-expand the F2FS on boot.