Modifying initrd/initramfs

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

I am new to low level linux so please excuse an ignorance in my questions.

Problem:
I need to enforce the check of a hardware security module, as early in the boot process as possible to prevent the machine from turning on unless an approved hardware key is plugged in USB.

My understanding is that initrd and initramfs were created precisely for this problem (being able to handle crypto operations before the drives can even be mounted and the OS Kernel loaded). They are essentially the same thing and initrd is the “old way” and initramfs is the “new way” with some minor differences in how the interact (using block devices or not which is irrelevant here, i don’t care about performance between the two).

Digging through the Jetpack 5.1.3 Linux_for_Tegra/ source, i found what i believe is the correct file to edit under: `~/Linux_for_Tegra/bootloader/l4t_initrd.img.

When i decompress this with unmkinitramfs i am presented with what appears to be a linux file system, and an init file which is doing all the “pre-boot verification” like checking is the NVME is LUKS encrypted, unlocking it and mounting so you can access the OS.

#!/bin/bash

# Copyright (c) 2014-2023, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of NVIDIA CORPORATION nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
initrd_dir=/mnt/initrd;
dhclient_flag="true";
count=0;

echo "Starting L4T initial RAM disk" > /dev/kmsg;

#Mount procfs, devfs, sysfs and debugfs
mount -t proc proc /proc
if [ $? -ne 0 ]; then
	echo "ERROR: mounting proc fail..." > /dev/kmsg;
	exec /bin/bash;
fi;
mount -t devtmpfs none /dev
if [ $? -ne 0 ]; then
	echo "ERROR: mounting dev fail..." > /dev/kmsg;
	exec /bin/bash;
fi;
mount -t sysfs sysfs /sys
if [ $? -ne 0 ]; then
	echo "ERROR: mounting sys fail..." > /dev/kmsg;
	exec /bin/bash;
fi;
mount -t debugfs none /sys/kernel/debug/
if [ $? -ne 0 ]; then
	echo "ERROR: mounting debugfs fail..." > /dev/kmsg;
	exec /bin/bash;
fi;

# create reboot command based on sysrq-trigger
if [ -e "/proc/sysrq-trigger" ]; then
	echo -e "#!/bin/bash \necho b > /proc/sysrq-trigger;" > /sbin/reboot;
	chmod 755 /sbin/reboot;
fi;

dev_regex='root=\/dev\/[abcdefklmnpsv0-9]*'
uuid_regex='root=PARTUUID=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
ext4uuid_regex='root=UUID=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
rootdev="$(cat /proc/cmdline | grep -oE "\<${dev_regex}|${uuid_regex}|${ext4uuid_regex}\>" | tail -1)"
if [ "${rootdev}" != "" ]; then
	if [[ "${rootdev}" =~ "UUID" ]]; then
		rootdev=$(echo "${rootdev}" | sed -ne "s/root=\(.*\)/\1/p")
	else
		rootdev=$(echo "${rootdev}" | sed -ne "s/root=\/dev\/\(.*\)/\1/p")
	fi
	echo "Root device found: ${rootdev}" > /dev/kmsg;
fi

cd /usr/sbin;
ln -s /bin/kmod depmod
depmod -a
cd /

rootfs_is_encrypted=0
if [[ "${rootdev}" == PARTUUID* || "${rootdev}" == UUID* ]]; then
	if [ -e "/etc/crypttab" ]; then
		# Handle encrypted rootfs
......

Steps I have Taken:

  • I have modified this init file with my code to check for the hardware key. (already tested it working in a simple .sh script locally)
  • I installed the packages that would be required to import into the initrd image filesystem on another Jetson (to ensure they were ARM and not AMD packages, did not want to deal with QEMU on a AMD host machine).
  • I then used dpkg -L and ldd to list all the dependencies and identify all the source files i would need to copy over into the initrd image for my code to work.
  • Copied some files over to /lib/aarch64-linux-gnu/ and /usr/bin/
  • Re-compressed the image find . | cpio -H newc -o | gzip -9 > ../l4t_initrd-new.img and moved it back into ~/Linux_for_Tegra/bootloader/l4t_initrd.img
  • Flash using these steps:
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 --no-flash --showlogs -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" jetson-orin-nano-devkit internal

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 300GiB --external-only --append --network usb0 jetson-orin-nano-devkit external

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

However, I am getting stuck during the last part of the flashing process, here is my UART and flash logs.

UART Output

[    7.307387] Root device found: initrd
insmod /lib/modules/5.10.120-tegra/kernel/drivers/net/ethernet/realtek/r8168.ko 
[    7.315509] r8168 0008:01:00.0: Adding to iommu group 12
[    7.315936] r8168 Gigabit Ethernet driver 8.051.02-NAPI loaded
[    7.316186] r8168 0008:01:00.0: enabling device (0000 -> 0003)
[    7.334185] r8168: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[    7.336548] r8168  Copyright (C) 2022 Realtek NIC software team <nicfae@realtek.com> 
[    7.336548]  This program comes with ABSOLUTELY NO WARRANTY; for details, please see <http://www.gnu.org/licenses/>. 
[    7.336548]  This is free software, and you are welcome to redistribute it under certain conditions; see <http://www.gnu.org/licenses/>. 
[    7.338603] Mount initrd as rootfs and enter recovery mode
enable_remote_access
enable remote access
insmod /lib/modules/5.10.120-tegra/kernel/drivers/mtd/mtd.ko 
insmod /lib/modules/5.10.120-tegra/kernel/drivers/mtd/devices/qspi_mtd.ko 
[    7.362803] qspi_mtd spi6.0: MX25U51279G (65536 Kbytes)
[    7.364710] qspi_mtd spi6.0: mtd .name = spi6.0, .size = 0x4000000 (64MiB) .erasesize = 0x00010000 (64KiB) .numeraseregions = 0
[    7.381706] 1 fixed-partitions partitions found on MTD device spi6.0
[    7.382721] Creating 1 MTD partitions on "spi6.0":
[    7.387732] 0x000000000000-0x000004000000 : "Whole_flash0"
insmod /lib/modules/5.10.120-tegra/kernel/drivers/spi/spi-tegra210-qspi.ko 
insmod /lib/modules/5.10.120-tegra/kernel/drivers/hwmon/pwm-fan.ko 
[    7.420337] using random self ethernet address
[    7.420469] using random host ethernet address
[    7.424089] Mass Storage Function, version: 2009/09/11
[    7.424244] LUN: removable file: (no medium)
[    7.426034] LUN: removable file: (no medium)
[    7.430256] LUN: removable file: (no medium)
[    7.434974] LUN: removable file: (no medium)
Add /dev/nvme0n1
[    7.442123] rndis0: HOST MAC 26:c1:9c:5d:46:3b
[    7.444295] rndis0: MAC c2:27:8c:b2:fc:6c
[    7.449358] tegra-xudc 3550000.xudc: EP 0 (type: ctrl, dir: out) enabled
[    7.727102] random: crng init done
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.0# [    7.907038] mmc1: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit
[    7.987849] tegra-xudc 3550000.xudc: EP 5 (type: intr, dir: in) enabled
[    7.988061] tegra-xudc 3550000.xudc: EP 3 (type: bulk, dir: in) enabled
[    7.988262] tegra-xudc 3550000.xudc: EP 2 (type: bulk, dir: out) enabled
[    7.988541] IPv6: ADDRCONF(NETDEV_CHANGE): rndis0: link becomes ready
[    7.988687] tegra-xudc 3550000.xudc: EP 7 (type: bulk, dir: in) enabled
[    7.988922] tegra-xudc 3550000.xudc: EP 4 (type: bulk, dir: out) enabled

Flash.log

Waiting for target to boot-up...
Waiting for device to expose ssh ......RTNETLINK answers: File exists
RTNETLINK answers: File exists
Run command: if [ -f /qspi/l4t_flash_from_kernel.sh ]; then USER=root /qspi/l4t_flash_from_kernel.sh --no-reboot --qspi-only ; fi on root@fe80::1%usb0
blockdev: cannot open /dev/sdc: No medium found
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /home/lenovo/nvidia/local_dev/Linux_for_Tegra/tools/kernel_flash/images/internal/flash.idx
Number of lines is 58
max_index=57
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
Flash index file is /qspi/internal/flash.idx
Number of lines is 58
max_index=57
[ 0]: l4t_flash_from_kernel: Starting to flash to qspi
QSPI storage size: 67108864 bytes.
[ 0]: l4t_flash_from_kernel: Successfully create gpt for emmc
Run command: partprobe /dev/mmcblk0 on root@fe80::1%usb0
Error: Could not stat device /dev/mmcblk0 - No such file or directory.