Modifying initrd/initramfs

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
1

I am new to low level linux so please excuse an ignorance in my questions.

Problem:
I need to enforce the check of a hardware security module, as early in the boot process as possible to prevent the machine from turning on unless an approved hardware key is plugged in USB.

My understanding is that initrd and initramfs were created precisely for this problem (being able to handle crypto operations before the drives can even be mounted and the OS Kernel loaded). They are essentially the same thing and initrd is the “old way” and initramfs is the “new way” with some minor differences in how the interact (using block devices or not which is irrelevant here, i don’t care about performance between the two).

Digging through the Jetpack 5.1.3 Linux_for_Tegra/ source, i found what i believe is the correct file to edit under: `~/Linux_for_Tegra/bootloader/l4t_initrd.img.

When i decompress this with unmkinitramfs i am presented with what appears to be a linux file system, and an init file which is doing all the “pre-boot verification” like checking is the NVME is LUKS encrypted, unlocking it and mounting so you can access the OS.

#!/bin/bash

# Copyright (c) 2014-2023, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of NVIDIA CORPORATION nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
initrd_dir=/mnt/initrd;
dhclient_flag="true";
count=0;

echo "Starting L4T initial RAM disk" > /dev/kmsg;

#Mount procfs, devfs, sysfs and debugfs
mount -t proc proc /proc
if [ $? -ne 0 ]; then
	echo "ERROR: mounting proc fail..." > /dev/kmsg;
	exec /bin/bash;
fi;
mount -t devtmpfs none /dev
if [ $? -ne 0 ]; then
	echo "ERROR: mounting dev fail..." > /dev/kmsg;
	exec /bin/bash;
fi;
mount -t sysfs sysfs /sys
if [ $? -ne 0 ]; then
	echo "ERROR: mounting sys fail..." > /dev/kmsg;
	exec /bin/bash;
fi;
mount -t debugfs none /sys/kernel/debug/
if [ $? -ne 0 ]; then
	echo "ERROR: mounting debugfs fail..." > /dev/kmsg;
	exec /bin/bash;
fi;

# create reboot command based on sysrq-trigger
if [ -e "/proc/sysrq-trigger" ]; then
	echo -e "#!/bin/bash \necho b > /proc/sysrq-trigger;" > /sbin/reboot;
	chmod 755 /sbin/reboot;
fi;

dev_regex='root=\/dev\/[abcdefklmnpsv0-9]*'
uuid_regex='root=PARTUUID=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
ext4uuid_regex='root=UUID=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
rootdev="$(cat /proc/cmdline | grep -oE "\<${dev_regex}|${uuid_regex}|${ext4uuid_regex}\>" | tail -1)"
if [ "${rootdev}" != "" ]; then
	if [[ "${rootdev}" =~ "UUID" ]]; then
		rootdev=$(echo "${rootdev}" | sed -ne "s/root=\(.*\)/\1/p")
	else
		rootdev=$(echo "${rootdev}" | sed -ne "s/root=\/dev\/\(.*\)/\1/p")
	fi
	echo "Root device found: ${rootdev}" > /dev/kmsg;
fi

cd /usr/sbin;
ln -s /bin/kmod depmod
depmod -a
cd /

rootfs_is_encrypted=0
if [[ "${rootdev}" == PARTUUID* || "${rootdev}" == UUID* ]]; then
	if [ -e "/etc/crypttab" ]; then
		# Handle encrypted rootfs
......

Steps I have Taken:

  • I have modified this init file with my code to check for the hardware key. (already tested it working in a simple .sh script locally)
  • I installed the packages that would be required to import into the initrd image filesystem on another Jetson (to ensure they were ARM and not AMD packages, did not want to deal with QEMU on a AMD host machine).
  • I then used dpkg -L and ldd to list all the dependencies and identify all the source files i would need to copy over into the initrd image for my code to work.
  • Copied some files over to /lib/aarch64-linux-gnu/ and /usr/bin/
  • Re-compressed the image find . | cpio -H newc -o | gzip -9 > ../l4t_initrd-new.img and moved it back into ~/Linux_for_Tegra/bootloader/l4t_initrd.img
  • Flash using these steps:
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 --no-flash --showlogs -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" jetson-orin-nano-devkit internal

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 300GiB --external-only --append --network usb0 jetson-orin-nano-devkit external

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

However, I am getting stuck during the last part of the flashing process, here is my UART and flash logs.

UART Output

[    7.307387] Root device found: initrd
insmod /lib/modules/5.10.120-tegra/kernel/drivers/net/ethernet/realtek/r8168.ko 
[    7.315509] r8168 0008:01:00.0: Adding to iommu group 12
[    7.315936] r8168 Gigabit Ethernet driver 8.051.02-NAPI loaded
[    7.316186] r8168 0008:01:00.0: enabling device (0000 -> 0003)
[    7.334185] r8168: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[    7.336548] r8168  Copyright (C) 2022 Realtek NIC software team <nicfae@realtek.com> 
[    7.336548]  This program comes with ABSOLUTELY NO WARRANTY; for details, please see <http://www.gnu.org/licenses/>. 
[    7.336548]  This is free software, and you are welcome to redistribute it under certain conditions; see <http://www.gnu.org/licenses/>. 
[    7.338603] Mount initrd as rootfs and enter recovery mode
enable_remote_access
enable remote access
insmod /lib/modules/5.10.120-tegra/kernel/drivers/mtd/mtd.ko 
insmod /lib/modules/5.10.120-tegra/kernel/drivers/mtd/devices/qspi_mtd.ko 
[    7.362803] qspi_mtd spi6.0: MX25U51279G (65536 Kbytes)
[    7.364710] qspi_mtd spi6.0: mtd .name = spi6.0, .size = 0x4000000 (64MiB) .erasesize = 0x00010000 (64KiB) .numeraseregions = 0
[    7.381706] 1 fixed-partitions partitions found on MTD device spi6.0
[    7.382721] Creating 1 MTD partitions on "spi6.0":
[    7.387732] 0x000000000000-0x000004000000 : "Whole_flash0"
insmod /lib/modules/5.10.120-tegra/kernel/drivers/spi/spi-tegra210-qspi.ko 
insmod /lib/modules/5.10.120-tegra/kernel/drivers/hwmon/pwm-fan.ko 
[    7.420337] using random self ethernet address
[    7.420469] using random host ethernet address
[    7.424089] Mass Storage Function, version: 2009/09/11
[    7.424244] LUN: removable file: (no medium)
[    7.426034] LUN: removable file: (no medium)
[    7.430256] LUN: removable file: (no medium)
[    7.434974] LUN: removable file: (no medium)
Add /dev/nvme0n1
[    7.442123] rndis0: HOST MAC 26:c1:9c:5d:46:3b
[    7.444295] rndis0: MAC c2:27:8c:b2:fc:6c
[    7.449358] tegra-xudc 3550000.xudc: EP 0 (type: ctrl, dir: out) enabled
[    7.727102] random: crng init done
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.0# [    7.907038] mmc1: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit
[    7.987849] tegra-xudc 3550000.xudc: EP 5 (type: intr, dir: in) enabled
[    7.988061] tegra-xudc 3550000.xudc: EP 3 (type: bulk, dir: in) enabled
[    7.988262] tegra-xudc 3550000.xudc: EP 2 (type: bulk, dir: out) enabled
[    7.988541] IPv6: ADDRCONF(NETDEV_CHANGE): rndis0: link becomes ready
[    7.988687] tegra-xudc 3550000.xudc: EP 7 (type: bulk, dir: in) enabled
[    7.988922] tegra-xudc 3550000.xudc: EP 4 (type: bulk, dir: out) enabled

Flash.log

Waiting for target to boot-up...
Waiting for device to expose ssh ......RTNETLINK answers: File exists
RTNETLINK answers: File exists
Run command: if [ -f /qspi/l4t_flash_from_kernel.sh ]; then USER=root /qspi/l4t_flash_from_kernel.sh --no-reboot --qspi-only ; fi on root@fe80::1%usb0
blockdev: cannot open /dev/sdc: No medium found
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /home/lenovo/nvidia/local_dev/Linux_for_Tegra/tools/kernel_flash/images/internal/flash.idx
Number of lines is 58
max_index=57
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
Flash index file is /qspi/internal/flash.idx
Number of lines is 58
max_index=57
[ 0]: l4t_flash_from_kernel: Starting to flash to qspi
QSPI storage size: 67108864 bytes.
[ 0]: l4t_flash_from_kernel: Successfully create gpt for emmc
Run command: partprobe /dev/mmcblk0 on root@fe80::1%usb0
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
3

Hi brian178,

Are you using the devkit or custom board for Orin Nano?
What’s the physical size of your NVMe SSD?

Would it work if you don’t have any modification in init file?

Please share the full flash log for further check.

4

I was able to get it working, there were file permission issues with some of the binaries in the new image i made. Once i chmod 755 the new binaries i added to initramfs image it worked great.

Some things i learned which will hopefully help others:

How to make your logs in the init script show up in UART so you can debug the boot process:

echo "this message will show in UART debug logs" > /dev/kmsg;

The correct image to decompress and modify is:
/Linux_for_Tegra/bootloader/l4t_initrd.img

Use this command to decompress the image so you have a file system you can update (make sure you use sudo so file ownership and permissions are retained or this will cause major issue and kernel will not load):

mkdir l4t_initrd_decomp && sudo unmkinitramfs ../Linux_for_Tegra/bootloader/l4t_initrd.img l4t_initrd_decomp

Install whatever packages you need on a host jetson system (ensures they are already ARM64 packages). Then use dpkg -L and ldd to find all the dependency files the apps require to run. Copy over these files to the matching paths in the ./l4t_initrd_decomp folder (likely will be /lib/ and /usr/bin/ paths you need to copy stuff to)

MAKE SURE YOU APPLY 755 PERMISSIONS TO ANY NEW FILE YOU ADD!!!
Example: sudo chmod 755 libudev.so.1 libusb-1.0.so.0

Use this command to re-compress the image, then put it back into the bootloader folder and build your images and flash:

cd ~/nvidia/local_dev/Linux_for_Tegra/l4t_initrd_decomp/ && find . | cpio -H newc -o | gzip -9 > ../../l4t_initrd-new.img

5

Thanks for sharing them.