Xavier NX External Disk Encryption

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
, ,
1

Using Jetpack 4.6.1 with L4T 32.7.1, I am trying to enable disk encryption on a jetson xavier nx (production with onboard eMMC) with the rootfs existing on an external drive (an SD card), however using l4t_initrd_flash.sh as described in both the online docs and README_initrd_flash.txt, does not work. I’ve created the following helper script for this task:

#!/bin/bash                                                                                                               
sudo ROOTFS_ENC=1 bash -x tools/kernel_flash/l4t_initrd_flash.sh --no-flash --showlogs \
        -u /mnt/keys//pkc.pem -v /mnt/keys//sbk \
        -p "-i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash" \
        jetson-xavier-nx-devkit-qspi internal |& tee qspi.log
read -rp "Recovery"
sudo ROOTFS_ENC=1 bash -x tools/kernel_flash/l4t_initrd_flash.sh --no-flash --showlogs --append \
        -u /mnt/keys//pkc.pem -v /mnt/keys//sbk \
        "-p -i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash" \
        -c tools/kernel_flash/flash_d4t_sdcard_512GB_rootfs_enc.xml \
        --external-device mmcblk1p1 --external-only -S 100GiB jetson-xavier-nx-devkit-emmc mmcblk1p1 \
        |& tee external.log
read -rp "Recovery"
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --flash-only |& tee flash.log

Attached are the logs from each of the three steps:
qspi.log (199.6 KB)
flash.log (22.8 KB)
external.log (157.0 KB)

I’ve seen various errors here including: scripts replacing my external storage targets with internal targets, scripts denying that external disk encryption is supported, flashing succeeding but device is failing to boot (with multiple different failures).

I am at a loss at what to try next, all the Nvidia docs point to this function being supported but I cannot seem to get it to work.

Additional background:

I am using boards that have secure boot enabled (SKB+PKC) and all KEK fuses burned.

I have been able to successfully get disk encryption working using the internal eMMC by: changing the eks.img, flashing (with: sudo ROOTFS_ENC=1 ./flash.sh --showlogs -u /mnt/keys/pkc.pem -v /mnt/keys/sbk -i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash jetson-xavier-nx-devkit-emmc mmcblk0p1), booted and proved disk encryption is enabled.

I have digested other forum posts for information as well:

https://forums.developer.nvidia.com/t/disk-encryption-of-external-sda-in-jatson-agx-xavier-rs32-7-1-js4-6-1/211094/60?page=2
https://forums.developer.nvidia.com/t/encrypted-filesystem-works-on-32-6-1-but-not-on-32-5-1/205243
Why the initrd_flash can encrypt the disk without SBKPKC mode - #16 by QOP

The current setup I am using to flash and boot from an external drive (SD card) with secure boot is as follows:

# Configure jetson to boot from sd card - no rootfs flash
sudo ./flash.sh -u /mnt/keys/pkc.pem -v /mnt/keys/sbk jetson-xavier-nx-devkit-emmc mmcblk1p1
# Reboot to RCM
# Flash rootfs to sd card
bash -x tools/kernel_flash/l4t_initrd_flash.sh --showlogs \
    -u /mnt/keys/pkc.pem -v /mnt/keys/sbk \
    --external-device mmcblk1p1 -S 470GiB -c tools/kernel_flash/flash_l4t_sdcard_512GB.xml \
    --external-only jetson-xavier-nx-devkit-emmc mmcblk1p1
3

Hi,
This topic is close to your use-case:
Why the initrd_flash can encrypt the disk without SBKPKC mode

So you follow it step by step and still fail?

4

Correct, I followed that topic all the way through and was still unable to get my use case working.

Personally, I found that thread hard to follow because (to my interpretation) it started out as addressing external disk encryption but then the later suggestions solely dealt with using flash.sh for flashing, which does not appear to be able to flash externally attached disks (with or without disk encryption). To be fair though, that specific thread was a huge help in getting disk encryption working on the internal eMMC.

5

@DaneLLL is there any better documentation for getting external disk encryption setup?

6

Hi,
Do you try it on Xavier NX developer kit+NVMe SSD? You mention the rootfs is on SD card so looks like you are using your own custom board. This can be specific to the setup.

7

I have not tried that. This is indeed a custom carrier board that supports the SD card, however I am able to use the Nvidia tool chain to flash and boot from the SD card on the carrier board without disk encryption.

I might be able to try using an NVMe drive on our carrier board, however this would only be acceptable for testing the procedure. Realistically, I need to get encryption working on the SD card.

8

I look through your flash log and seems that the qspi cannot be found when flashing through initrd. This is probably due to it being a custom board so the kernel dtb does not correctly expose the qspi? A workaround for disk encryption I can think of is:

First: flash to qspi using flash.sh

sudo ./flash.sh -u /mnt/keys/pkc.pem -v /mnt/keys/sbk -i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash jetson-xavier-nx-devkit-qspi mmcblk1p1

Second flash to mmcblk1p1 only using initrd_flash:

sudo ROOTFS_ENC=1 bash -x tools/kernel_flash/l4t_initrd_flash.sh  --showlogs \
        -u /mnt/keys/pkc.pem --erase-all -v /mnt/keys/sbk \
        -p "-i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash" \
        -c tools/kernel_flash/flash_d4t_sdcard_512GB_rootfs_enc.xml \
        --external-device mmcblk1p1 --external-only -S 100GiB jetson-xavier-nx-devkit-emmc mmcblk1p1
9

I am wondering when the board boot, can you find the /dev/mtd0 node in devfs?

10

@lhoang when i try the above flashing commands, it sends the board into a boot loop - namely the SoC cannot find the kernel. Attached are the logs from the flash steps
flash_sdcard.log (187.0 KB)
flash_qspi.log (99.6 KB)
and logs from the serial console
serial.log (32.3 KB)
Relevant snip from serial log

[0010.300] W> No valid slot number is found in scratch register
[0010.305] W> Return default slot: _a
[0010.309] I> A/B: bin_type (37) slot 0
[0010.312] I> Loading kernel from partition
[0010.316] E> Cannot find partition kernel
[0010.320] E> Cannoÿâ

The main issue I am seeing in the logs (flash_sdcard.log) is that it switches the target device from mmcblk1 to internal, snip:

************************************
*                                  *
*  Step 1: Generate flash packages *
*                                  *
************************************
Create folder to store images to flash
Generate images to be flashed
BOOTDEV=mmcblk1p1 /home/jhempsey/workspace/d4t/work/Linux_for_Tegra/flash.sh --no-flash --sign --external-device -c "tools/kernel_flash/flash_d4t_sdcard_512GB_rootfs_enc.xml" -S "100GiB" -u "/mnt/keys/pkc.pem" -v "/mnt/keys/sbk" -i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash jetson-xavier-nx-devkit-emmc mmcblk1p1

###############################################################################
# L4T BSP Information:
# R32 , REVISION: 7.1
###############################################################################
Change device boot from mmcblk1p1 to internal

After a “normal” flash (i.e. recovering the board so it is operational), I do not see /dev/mtd0, likely we will need to fix the kernel DTB file for this as you indicated. Does this actually need to be remedied for external encryption to work properly?

11

Hi,
Please try to generate the images first and then flash. Check if it works in this method. May refer to
How to created an encrypted disk partition for data storage on Xavier NX - #3 by DaneLLL

12

No, actually. I take a look at your boot log and it seems that external disk encryption does not work because cboot cannot detect your sd card to boot from your sd card. Currently for our flash configuration with disk encryption, disk encryption only work if cboot can boot from the same storage device where the encrypted partition is located. I think you can work around this by doing the following

First flash to external sd card

sudo ROOTFS_ENC=1 bash -x tools/kernel_flash/l4t_initrd_flash.sh  --showlogs \
        -u /mnt/keys/pkc.pem --erase-all -v /mnt/keys/sbk \
        -p "-i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash" \
        -c tools/kernel_flash/flash_d4t_sdcard_512GB_rootfs_enc.xml \
        --external-device mmcblk1p1 --external-only -S 100GiB jetson-xavier-nx-devkit-emmc mmcblk1p1

The above command should generate a file called “system_boot.img”, If your “flash_d4t_sdcard_512GB_rootfs_enc.xml” is similar to “tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml” example. this is the boot partition that will mount the encrypted partition. You can then edit the file bootloader/t186ref/cfg/flash_l4t_t194_spi_emmc_p3668.xml so that the APP partition is “system_boot.img”. After that you can flash everything along with the boot partition onto the emmc

sudo NO_ROOTFS=1 ./flash.sh -u /mnt/keys/pkc.pem -v /mnt/keys/sbk -i /mnt/keys/ekb.key --user_key /mnt/keys/user_key_flash jetson-xavier-nx-devkit-emmc mmcblk1p1

After this, the device should boot from the internal emmc and mount and unlock the external encrypted partition.

1 Like
13

Crazy, this was the key. It actually worked. Thank you!

I thought I would also have to configure APPSIZE for the new system_boot.img but the flash fails if I do so. Using APPSIZE looks like it correctly auto-detects the size (and of course flashes and boots).

I do have a couple of follow on questions that hopefully you could answer to go along with this:

  1. Based on these findings and your suggestions (not looking to do this immediately, just looking to understand), does that mean that if I want to boot from the SD card in entirety, I need to modified cboot for this purpose? Or is this simply not possible?
  2. I played around with the eMMC storage a bit after flash and boot, by using this work around is it possible to navigate a bit outside of the documentation and eliminate some of the required partitions (since they’re also on the SD card) so that I can leverage some of the space on the eMMC? Or are all of the marked required partitions in the documentation still required in this setup?

Again, huge thank you for all of your help

14
  1. I think you could try to modify cboot yes.
  2. Yeah, you can remove the some of required partitions on emmc if they are on sd card.
15
  1. Actually, I asked around and it seems that if you requires feature such as partition based OTA, it is better if you have the required partitions on emmc. In which case, I think you could remove those partitions from the external storage device and/or resize APP partition on the emmc to be smaller similar to the size in flash_d4t_sdcard_512GB_rootfs_enc.xml :
        <partition name="APP" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 419430400 </size> <!-- this size -->
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <align_boundary> 4096 </align_boundary>
            <unique_guid> APPUUID </unique_guid>
            <filename> system_boot.img </filename>
            <description> **Required.** Contains the boot partition. This partition must be defined
              after `primary_GPT` so that it can be accessed as the fixed known special device
              `/dev/mmcblk0p1`. </description>
        </partition>
1 Like
16

Awesome, thank you for all the information. I think I can tinker with the partition sizes then if I need to go that route. Again, this was a huge help and I can’t thank you enough.

18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.