Hello,
I am working on a procedure for cloning Jestson Orin Nano with encrypted NVMe using mass flash.
I am currently running on JP6.0 and using the devkit board.
These are the steps I am doing:
- Generate ekb partition, with all keys zeroed except for disk encryption
sudo ./source/optee/samples/hwkey-agent/host/tool/gen_ekb/example.sh
sudo ./flash.sh --no-flash -k A_eks jetson-orin-nano-devkit internal
sudo cp bootloader/eks_t234_sigheader.img.encrypt tools/kernel_flash/images/internal/
- Flash QSPI and NVMe
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal
sudo ROOTFS_ENC=1 EXT_NUM_SECTORS=468846000 ./tools/kernel_flash/l4t_initrd_flash.sh -S 221GiB ``-p "--generic-passphrase"`` --massflash 4 --showlogs --no-flash --external-device nvme0n1p1 -i ./ekb.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-orin-nano-devkit external
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 2
- Connect to flashed Jetson and add all the necessary applications and files.
- Perform backup
sudo ./tools/backup_restore/l4t_backup_restore.sh -b -e nvme0n1 -c jetson-orin-nano-devkit
- Flash from backup.
sudo ROOTFS_ENC=1 EXT_NUM_SECTORS=468846000 ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 --massflash 2 -i ./ekb.key ''-p "--generic-passphrase"'' jetson-orin-nano-devkit nvme0n1
- Try to flash the Jetson
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 2
This fails with:
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Serial Number: 1421524215215
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 3
max_index=2
writing item=1, 1:3:primary_gpt,0,20480,gptmbr.img,20480,fixed-<reserved>-0,9ee1e8c783b80cf6c0d51b2b21a0af01c38c51c0
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
Flash failure
Either the device cannot mount the NFS server on the host or a flash command has failed. Debug log saved to /tmp/tmp.hsrzB1eu5e. You can access the target's terminal through "sshpass -p root ssh root@fc00:1:1:0::2"
Cleaning up...
flash_3-1_0_20241216-223610.log (8.8 KB)
uart-out-flash-only.txt (67.4 KB)
tmp.hsrzB1eu5e.txt (5.3 KB)
What am I doing wrong? Is there a better way to clone devices and NVMe’s?
I really appreciate any help you can provide.
hello shai.segev,
please see-also Topic 291335 for reference.
Hello @JerryChang,
Thanks for the response. I’ll review my procedure and try again.
The thing is that I need to flash the Jetson from backup using the --use-backup-image
,
Is this supported? Also, how should I create the “golden” image? Can it be from an unencrypted device, or should it also be encrypted with a generic key?
Thanks,
Shai
hello shai.segev,
since it’s by default to unlock the encrypted root device with the per-device unique passphrase, (i.e. ECID)
you’ll need to enable generic-passphrase for your massflash use-case.
besides, you may see-also… $OUT/Linux_for_Tegra/tools/backup_restore/README_backup_restore.txt
Thanks for your response. I went through the documentation again and found a slight problem. However, I am still unable to get the procedure to duplicate units to work.
I don’t get any errors during the flashing, but it takes a very short time to flash, which may indicate that the flashing is not happening or is partial.
These are the steps I do:
-
Backup a device using:
sudo ./tools/backup_restore/l4t_backup_restore.sh -b -e nvme0n1 -c jetson-orin-nano-devkit
-
Replace jetson and NVME on the board.
-
Flash qspi (with --no-flash):
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal
qspi_2024-12-22-2.log (190.1 KB)
-
Flash nvme (with --no-flash):
sudo ROOTFS_ENC=1 EXT_NUM_SECTORS=468846000 ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 --massflash 2 -i ./ekb.key ``-p "--generic-passphrase"`` jetson-orin-nano-devkit external
from_back_up2024-12-22-2.log (19.5 MB)
-
Flash the Jetson (with --flash-only)
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 4
flash_3-1_0_20241222-165842.log (39.2 KB)
uart_out.txt (130.7 KB)
hello shai.segev,
please also check whether you’ve an initrd flash image created.
besides, please also narrow down the issue by excluding disk encryption for testing.
Hi @JerryChang ,
I checked the directories.
the mfi_jetson-orin-nano-nvme-nt is only 4.2Gb, which is relatively small considering that my backed-up image has docker images and data of about 8GB. The directory tools/backup_restore/images contain 15G of data, which makes more sense.
Without encryption, it worked, but I am going to repeat the test to make sure I haven’t missed anything
Thanks
thanks, looking forward to your test results.
Hi @JerryChang ,
I tested mass flash without encryption. This time, the mfi_jetson-orin-nano-devkit
directory size is 37G, which makes more sense.
However, when trying to flash I am getting the error
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
This are the steps I did:
sudo ./tools/backup_restore/l4t_backup_restore.sh -b -e nvme0n1 -c jetson-orin-nano-devkit
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 --massflash 4 jetson-orin-nano-devkit nvme0n1p1
sudo systemctl start nfs-kernel-server
sudo systemctl stop udisks2
/opt/nvidia/Linux_for_Tegra$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 4 --network usb0
massflash-use-backup.log (19.6 MB)
backup-no-encrypt.log (312.1 KB)
flash_3-1_0_20241223-173902.log (8.7 KB)
hello shai.segev,
you may flashing multiple devices once you’ve mfi package created.
for instance,
$ cd $OUT/Linux_for_Tegra/mfi_jetson-orin-nano-devkit
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --massflash 2 --network usb0 --showlogs --flash-only
anyways,
please give it another try by adding some delay (i.e. sleep for 0.5-sec) within l4t_initrd_flash.sh
for testing.
for instance,
for devpath in "${devpaths[@]}"; do
...
if [ "${max_massflash}" -eq 1 ]; then
break
fi
instance=$((instance + 1))
+ sleep 0.5
done
Hello @JerryChang ,
I followed your instructions, but I am still getting the same error.
Regards
# Entry added by NVIDIA initrd flash tool
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/tools/kernel_flash/tmp 127.0.0.1(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
Export list for localhost:
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/tools/kernel_flash/tmp 127.0.0.1
**********************************************
* *
* Step 1: Build the flashing environment *
* *
**********************************************
Create flash environment 0
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/bootloader /opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit
Finish creating flash environment 0.
****************************************************
* *
* Step 2: Boot the device with flash initrd image *
* *
****************************************************
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0 /opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit
./tegraflash.py --bl uefi_jetson_with_dtb_sigheader.bin.encrypt --bct br_bct_BR.bct --securedev --bldtb tegra234-p3768-0000+p3767-0003-nv.dtb --applet rcm_2_encrypt.rcm --applet_softfuse rcm_1_encrypt.rcm --cmd "rcmboot" --cfg secureflash.xml --chip 0x23 --mb1_bct mb1_bct_MB1_sigheader.bct.encrypt --mem_bct mem_rcm_sigheader.bct.encrypt --mb1_cold_boot_bct mb1_cold_boot_bct_MB1_sigheader.bct.encrypt --mb1_bin mb1_t234_prod_aligned_sigheader.bin.encrypt --psc_bl1_bin psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --mem_bct_cold_boot mem_coldboot_sigheader.bct.encrypt --bins "psc_fw pscfw_t234_prod_sigheader.bin.encrypt; mts_mce mce_flash_o10_cr_prod_sigheader.bin.encrypt; tsec_fw tsec_t234_sigheader.bin.encrypt; mb2_applet applet_t234_sigheader.bin.encrypt; mb2_bootloader mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt; xusb_fw xusb_t234_prod_sigheader.bin.encrypt; pva_fw nvpva_020_sigheader.fw.encrypt; dce_fw display-t234-dce_sigheader.bin.encrypt; nvdec nvdec_t234_prod_sigheader.fw.encrypt; bpmp_fw bpmp_t234-TE950M-A1_prod_sigheader.bin.encrypt; bpmp_fw_dtb tegra234-bpmp-3767-0003-3509-a02_with_odm_sigheader.dtb.encrypt; rce_fw camera-rtcpu-t234-rce_sigheader.img.encrypt; ape_fw adsp-fw_sigheader.bin.encrypt; spe_fw spe_t234_sigheader.bin.encrypt; tos tos-optee_t234_sigheader.img.encrypt; eks eks_t234_sigheader.img.encrypt; kernel boot0.img; kernel_dtb tegra234-p3768-0000+p3767-0003-nv.dtb" --bct_backup --instance 3-1
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegraflash_internal.py:1272: SyntaxWarning: invalid escape sequence '\.'
patt = re.compile(".*(mbr|gpt).*\.bin")
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegraflash_internal.py:1388: SyntaxWarning: invalid escape sequence '\.'
patt = re.compile(".*(mbr|gpt).*\.bin")
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegraflash_internal.py:4587: SyntaxWarning: invalid escape sequence '\s'
m = re.search('bpmp_fw_dtb[\s]+([\w._-]+)', values['--bins'])
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegrasign_v3_internal.py:1124: SyntaxWarning: invalid escape sequence '\d'
m = re.search('Key size is (\d+)', ret_str)
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegrasign_v3_internal.py:1169: SyntaxWarning: invalid escape sequence '\d'
m = re.search('Key size is (\d+)', ret_str)
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegrasign_v3_internal.py:1527: SyntaxWarning: invalid escape sequence '\d'
re_string = 'kdf_args_' + temp_stem + '(\d).yaml'
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegraflash_impl_t234.py:1480: SyntaxWarning: invalid escape sequence '\s'
m = re.search('bpmp_fw_dtb[\s]+([\w._-]+)', values['--bins'])
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegraflash_impl_t234.py:2942: SyntaxWarning: invalid escape sequence '\.'
patt = re.compile(".*(mbr|gpt).*\.bin")
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit/temp_initrdflash/bootloader0/tegraflash_impl_t234.py:3004: SyntaxWarning: invalid escape sequence '\/'
info_print("Copying enc\/signed file in " + output_dir)
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
Entering RCM boot
[ 0.0433 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from --mb1_bin
[ 0.0433 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from --psc_bl1_bin
[ 0.0433 ] rcm boot with presigned binaries
[ 0.0440 ] tegrarcm_v2 --instance 3-1 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[ 0.0444 ] BR_CID: 0x80012344705DF1015C00000017FE81C0
[ 0.0568 ] Sending bct_br
[ 0.0667 ] Sending mb1
[ 0.0676 ] Sending psc_bl1
[ 0.0840 ] Sending bct_mb1
[ 0.0917 ] Generating blob for T23x
[ 0.0932 ] tegrahost_v2 --chip 0x23 0 --generateblob blob.xml blob.bin
[ 0.0935 ] The number of images in blob is 19
[ 0.0940 ] blobsize is 116395289
[ 0.0942 ] Added binary blob_uefi_jetson_with_dtb_sigheader.bin.encrypt of size 3563584
[ 0.1397 ] Added binary blob_pscfw_t234_prod_sigheader.bin.encrypt of size 310768
[ 0.1402 ] Added binary blob_mce_flash_o10_cr_prod_sigheader.bin.encrypt of size 187120
[ 0.1407 ] Added binary blob_tsec_t234_sigheader.bin.encrypt of size 176128
[ 0.1412 ] Added binary blob_applet_t234_sigheader.bin.encrypt of size 279616
[ 0.1416 ] Not supported type: mb2_applet
[ 0.1418 ] Added binary blob_mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt of size 439968
[ 0.1424 ] Added binary blob_xusb_t234_prod_sigheader.bin.encrypt of size 164864
[ 0.1429 ] Added binary blob_nvpva_020_sigheader.fw.encrypt of size 2164640
[ 0.1433 ] Added binary blob_display-t234-dce_sigheader.bin.encrypt of size 12065184
[ 0.1479 ] Added binary blob_nvdec_t234_prod_sigheader.fw.encrypt of size 294912
[ 0.1496 ] Added binary blob_bpmp_t234-TE950M-A1_prod_sigheader.bin.encrypt of size 1027008
[ 0.1501 ] Added binary blob_tegra234-bpmp-3767-0003-3509-a02_with_odm_sigheader.dtb.encrypt of size 204672
[ 0.1508 ] Added binary blob_camera-rtcpu-t234-rce_sigheader.img.encrypt of size 458096
[ 0.1513 ] Added binary blob_adsp-fw_sigheader.bin.encrypt of size 414960
[ 0.1518 ] Added binary blob_spe_t234_sigheader.bin.encrypt of size 270336
[ 0.1522 ] Added binary blob_tos-optee_t234_sigheader.img.encrypt of size 1633344
[ 0.1526 ] Added binary blob_eks_t234_sigheader.img.encrypt of size 9232
[ 0.1531 ] Added binary blob_boot0.img of size 92485632
[ 0.1996 ] Added binary blob_tegra234-p3768-0000+p3767-0003-nv.dtb of size 244121
[ 0.2498 ] tegrarcm_v2 --instance 3-1 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
[ 0.2501 ] BL: version 1.4.0.2-t234-54845784-08a4de08 last_boot_error: 0
[ 0.3534 ] Sending bct_mem
[ 0.3645 ] Sending blob
[ 5.3220 ] RCM-boot started
/opt/nvidia/Linux_for_Tegra/mfi_jetson-orin-nano-devkit
***************************************
* *
* Step 3: Start the flashing process *
* *
***************************************
chown: warning: '.' should be ':': ‘root.root’
chown: warning: '.' should be ':': ‘root.root’
Waiting for target to boot-up...
....
Waiting for target to boot-up...
Waiting for device to expose ssh ......Waiting for device to expose ssh ...Run command: flash on fc00:1:1:0::2
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Serial Number: 1421524215215
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 3
max_index=2
writing item=1, 1:3:primary_gpt,0,20480,gptmbr.img,20480,fixed-<reserved>-0,745eeb1557e66e15e3fc223ce67f7a17faaf4430
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
Flash failure
Either the device cannot mount the NFS server on the host or a flash command has failed. Debug log saved to /tmp/tmp.zleEtnE4v9. You can access the target's terminal through "sshpass -p root ssh root@fc00:1:1:0::2"
Cleaning up...
are you still working on JP-6.0 public release version?
is it possible for moving forward since Jetpack-6.1 is now available.
Hi @JerryChang ,
Thanks for your response. I tried 6.1 when it was released, but it did not work. Also, we spent a lot of time testing everything with 6.0, which we will have to repeat for 6.1.
I’d appreciate it if you could provide some fix/workaround for 6.0.
Thanks
hello shai.segev,
is this issue due to backup/restore/massflash? did you also verify disk encryption?
please see-also Topic 314134, there’re some changes of OEM_K1/K2 to derive a root key.
besides, we’ve test locally to confirm disk encryption is working.
here’re our test steps for your reference, i.e. Topic 314134, comment #29.
Hi @JerryChang,
Right now, I am at a point where I get the same error when there is no disk encryption.
So, I am not sure it is related.
My plan is to get Massflash to work on an insecure system, then adapt it to the encrypted system, and then to a secure boot system.
So, right now, I am trying to get a backup and massflash from the backup image to work on an unencrypted disk.
Thanks
hello shai.segev,
are you trying to re-flash backup image to another Orin NX device with an empty NVMe?
if yes…
please refer to Topic 311018 to apply the fixes for verification.
for instance,
diff --git a/scripts/backup-restore/nvrestore_partitions.sh b/scripts/backup-restore/nvrestore_partitions.sh
index 0cebae9..77d2cd5 100755
--- a/scripts/backup-restore/nvrestore_partitions.sh
+++ b/scripts/backup-restore/nvrestore_partitions.sh
@@ -54,6 +54,7 @@
BLOCK_DEVICE_LIST=("nvme0n1")
MMC_DEVICES=("3701" "3767-0005")
MODEL=$(tr -d '\0' < /proc/device-tree/compatible)
+BLOCK_SIZE=512
for MMC_DEVICE in "${MMC_DEVICES[@]}"
do
if echo "${MODEL}" | grep -q "${MMC_DEVICE}"; then
@@ -301,6 +302,16 @@
exit 1
fi
+declare -A able_to_delete
+
+for device in "${BLOCK_DEVICE_LIST[@]}"; do
+ if blkdiscard -f "/dev/${device}" &>/dev/null; then
+ able_to_delete["${device}"]="success"
+ else
+ able_to_delete["${device}"]=""
+ fi
+done
+
# The GPT must be the first partition flashed, so this block ensures that the
# GPT exists and is flashed first.
# shellcheck disable=SC2013
@@ -369,10 +380,14 @@
echo "${SCRIPT_NAME} Checksum of ${FIELDS[2]} does not match the checksum in the index file."
exit
fi
+ options=("status=progress" "bs=${BLOCK_SIZE}")
if [ "${FIELDS[2]}" = 'gpt_2' ]; then
- dd if="${FIELDS[1]}" of="/dev/${BLOCK_DEVICE}" bs=512 seek=$((FIELDS[3])) count=$((FIELDS[4]))
+ dd if="${FIELDS[1]}" of="/dev/${BLOCK_DEVICE}" "${options[@]}" seek=$((FIELDS[3])) count=$((FIELDS[4]))
else
- zstd -dc "${FIELDS[1]}" | dd of="/dev/${FIELDS[2]}" status=progress conv=sparse bs=512 seek=$((FIELDS[3])) count=$((FIELDS[4]))
+ if [ -n "${able_to_delete["${BLOCK_DEVICE}"]}" ]; then
+ options+=("conv=sparse")
+ fi
+ zstd -dc "${FIELDS[1]}" | dd of="/dev/${FIELDS[2]}" "${options[@]}" seek=$((FIELDS[3])) count=$((FIELDS[4]))
fi
fi
done
Hello @JerryChang
I just noticed I posted in the wrong section. We are working with Orin Nano.
I took the scripts under tools/backup_restore from JP6.1 and applied the fix you provided in the last message.
I am following “Workflow 3: To massflash the backup image” as follows:
sudo ./tools/backup_restore/l4t_backup_restore.sh -b -c jetson-orin-nano-devkit
back2901.txt (9.1 MB)
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 --massflash 2 jetson-orin-nano-devkit internal
use-backup-image.txt (19.6 MB)
cd mfi_jetson-orin-nano-devkit
(not in procedure but per your previous response)
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 2 --network usb0
flash_3-1_0_20241229-091125.log (20.2 KB)
- Try to boot the device… the device boot fails to find a valid boot partition and trying to boot from network.
What am I doing wrong?
Thanks
hello shai.segev,
please also adding -e
to specify the device for backup in the the script to creates a backup image.
i.e. $ sudo ./tools/backup_restore/l4t_backup_restore.sh -b -c -e nvme0n1 jetson-orin-nano-devkit
besides, let’s narrow down the issue with single device by using --massflash 1
.
Hi @JerryChang
I tried simple backup restore on NON encrypted devices.
sudo ./tools/backup_restore/l4t_backup_restore.sh -b -e nvme0n1 jetson-orin-nano-devkit
On the golden jetson and nvme.
Then take another jetson and nvme and try to restore on to it.
sudo ./tools/backup_restore/l4t_backup_restore.sh -r -e nvme0n1 jetson-orin-nano-devkit
After the restore is complete, the system will not boot.
If I replace just the jetson, it works ok with the replicated nvme
Is that because the -e or is it something else?
Also tried:
sudo ./tools/backup_restore/l4t_backup_restore.sh -b -c -e nvme0n1 jetson-orin-nano-devkit
backup-e-c-nvme.txt (102.5 KB)
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 --massflash 4 jetson-orin-nano-devkit internal
use-backup-image-massflash1.txt (19.5 MB)
Replace jetson and nvme:
cd mfi_jetson-orin-nano-devkit/
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 1 --network usb0
flash_3-1_0_20241230-205631.log (20.3 KB)
Still not working