Failing consecutively to extend APP partition for booting on NVME SSD

neuroSparK · May 29, 2024, 8:47pm

I am using a Jetson Xavier NX production module (16GB emmc) with Seeedstudio A302 carrier board. I have successfully flashed the emmc to Jetpack 5.1.2 with SDK manager. But I failed to flash and boot it from 128GB Twinmos NVME SSD.

Disk /dev/nvme0n1: 119.25 GiB, 128035676160 bytes, 250069680 sectors
Disk model: TwinMOS SSD                             
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Here is my attempts:

This command successfully flashed the NX and booted from Nvme SSD.

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-only -c ./tools/kernel_flash/flash_l4t_nvme.xml --external-device nvme0n1  --showlogs jetson-xavier-nx-devkit-emmc external

But the APP partition (rootfs) is only 12GB which is not enough for me. Thus I tried to flash with larger APP partition with this command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-only -c ./tools/kernel_flash/flash_l4t_nvme.xml --external-device nvme0n1 -S 100GiB --showlogs jetson-xavier-nx-devkit-emmc external

But it fails to flash with no space left message

tar: ./etc/ssl/certs/HARICA_TLS_RSA_Root_CA_2021.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/HARICA_TLS_RSA_Root_CA_2021.crt’: No space left on device
tar: ./etc/ssl/certs/SSL.com_EV_Root_Certification_Authority_RSA_R2.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/SSL.com_EV_Root_Certification_Authority_RSA_R2.crt’: No space left on device
tar: ./etc/ssl/certs/Entrust_Root_Certification_Authority_-_G4.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority_-_G4.crt’: No space left on device
tar: ./etc/ssl/certs/Go_Daddy_Root_Certificate_Authority_-_G2.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Go_Daddy_Root_Certificate_Authority_-_G2.crt’: No space left on device
tar: ./etc/ssl/certs/Entrust_Root_Certification_Authority_-_G2.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority_-_G2.crt’: No space left on device
tar: ./etc/ssl/certs/Atos_TrustedRoot_2011.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Atos_TrustedRoot_2011.crt’: No space left on device
tar: ./etc/ssl/certs/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.crt’: No space left on device
tar: ./etc/ssl/certs/DigiCert_Global_Root_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/DigiCert_Global_Root_CA.crt’: No space left on device
tar: ./etc/ssl/certs/GlobalSign_Root_CA_-_R3.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/GlobalSign_Root_CA_-_R3.crt’: No space left on device
tar: ./etc/ssl/certs/DigiCert_Assured_ID_Root_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/DigiCert_Assured_ID_Root_CA.crt’: No space left on device
tar: ./etc/ssl/certs/T-TeleSec_GlobalRoot_Class_2.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/T-TeleSec_GlobalRoot_Class_2.crt’: No space left on device
tar: ./etc/ssl/certs/OISTE_WISeKey_Global_Root_GB_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/OISTE_WISeKey_Global_Root_GB_CA.crt’: No space left on device
tar: ./etc/ssl/certs/E-Tugra_Global_Root_CA_ECC_v3.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/E-Tugra_Global_Root_CA_ECC_v3.crt’: No space left on device
tar: ./etc/ssl/certs/Comodo_AAA_Services_root.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Comodo_AAA_Services_root.crt’: No space left on device
tar: ./etc/ssl/certs/DigiCert_Global_Root_G3.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/DigiCert_Global_Root_G3.crt’: No space left on device
tar: ./etc/ssl/certs/TWCA_Root_Certification_Authority.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/TWCA_Root_Certification_Authority.crt’: No space left on device
tar: ./etc/ssl/certs/Certum_Trusted_Network_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA.crt’: No space left on device
tar: ./etc/ssl/certs/Certum_Trusted_Root_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Certum_Trusted_Root_CA.crt’: No space left on device
tar: ./etc/ssl/certs/NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/NetLock_Arany_=Class_Gold=_Főtanúsítvány.crt’: No space left on device
tar: ./etc/ssl/certs/Actalis_Authentication_Root_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Actalis_Authentication_Root_CA.crt’: No space left on device
tar: ./etc/ssl/certs/Baltimore_CyberTrust_Root.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Baltimore_CyberTrust_Root.crt’: No space left on device
tar: ./etc/ssl/certs/Security_Communication_RootCA2.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Security_Communication_RootCA2.crt’: No space left on device
tar: ./etc/ssl/certs/IdenTrust_Commercial_Root_CA_1.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/IdenTrust_Commercial_Root_CA_1.crt’: No space left on device
tar: ./etc/ssl/certs/Buypass_Class_2_Root_CA.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Buypass_Class_2_Root_CA.crt’: No space left on device
tar: ./etc/ssl/certs/COMODO_Certification_Authority.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/COMODO_Certification_Authority.crt’: No space left on device
tar: Exiting with failure status due to previous errors
[ 16]: l4t_flash_from_kernel: Failed to decompress APP image into /dev/sdi1
[ 16]: l4t_flash_from_kernel: Failed to write to APP
[ 16]: l4t_flash_from_kernel: Error flashing external device
Error flashing non-qspi storage

Cleaning up...

I also tried to set the partition size from the flash_l4t_nvme.xml like this :

<partition_layout version="01.00.0000">
    <device type="external" instance="0" sector_size="512" num_sectors="250069680">
        <partition name="master_boot_record" type="protective_master_boot_record">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 512 </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <description> **Required.** Contains protective MBR. </description>
        </partition>
        <partition name="primary_gpt" type="primary_gpt">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 19968 </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <description> **Required.** Contains primary GPT of the `nvme` device. All
              partitions defined after this entry are configured in the kernel, and are
              accessible by standard partition tools such as gdisk and parted. </description>
        </partition>
        <partition name="APP" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 200000000 </size> <!-- 100 GiB in sectors -->
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <align_boundary> 4096 </align_boundary>
            <percent_reserved> 0 </percent_reserved>
            <filename> APPFILE </filename>
            <unique_guid> APPUUID </unique_guid>
            <description> **Required.** Contains the rootfs. This partition must be defined
              after `primary_GPT` so that it can be accessed as the fixed known special device
              `/dev/nvme0n1p1`. </description>
----------------------
-----------------------
<partition name="UDA" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> -0 </size> <!-- Automatically takes all remaining space -->
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x808 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <description> **Required.** Automatically takes all remaining space on the device except that
              occupied by the `secondary_gpt` partition. Allocation attribute must be set to 0x808.
              May be mounted and used to store user data. </description>
        </partition>
        <partition name="secondary_gpt" type="secondary_gpt">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 0xFFFFFFFFFFFFFFFF </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <description> **Required.** Contains secondary GPT of the `sdcard`
              device. </description>
        </partition>
    </device>
</partition_layout>

I made change to the APP partition and UDA partition so that the APP partition takes 100GB space. But still no way it’s flashing.

Please help me overcoming this frustrating situation.

KevinFFF · May 30, 2024, 2:54am

Hi neuroSparK,

If this command work with booting from NVMe and boot up successfully, have you tried enlarge the size for rootfs after boot up?

Do you have enough size on your host?

neuroSparK · May 30, 2024, 6:05am

Yes, I have 2TB space on the disk. I also have tried to enlarge the size for rootfs after boot up.

Device             Start      End  Sectors  Size Type
/dev/nvme0n1p1        40 29360167 29360128   14G Microsoft basic data
/dev/nvme0n1p2  29360168 29622311   262144  128M Microsoft basic data
/dev/nvme0n1p3  29622312 29623847     1536  768K Microsoft basic data
/dev/nvme0n1p4  29623848 29688615    64768 31.6M Microsoft basic data
/dev/nvme0n1p5  29688616 29950759   262144  128M Microsoft basic data
/dev/nvme0n1p6  29950760 29952295     1536  768K Microsoft basic data
/dev/nvme0n1p7  29952296 30017063    64768 31.6M Microsoft basic data
/dev/nvme0n1p8  30017064 30180903   163840   80M Microsoft basic data
/dev/nvme0n1p9  30180904 30181927     1024  512K Microsoft basic data
/dev/nvme0n1p10 30181928 30386727   204800  100M Microsoft basic data
/dev/nvme0n1p11 30386728 30517799   131072   64M EFI System
/dev/nvme0n1p12 30517800 30681639   163840   80M Microsoft basic data
/dev/nvme0n1p13 30681640 30682663     1024  512K Microsoft basic data
/dev/nvme0n1p14 30682664 30813735   131072   64M Microsoft basic data
I have deleted the UDA partiton and there is now 104.55 GiB unallocated space.

but since the APP partition is on the leftmost and the unallocated space on the rightmost, it’s tough to merge the APP partition to the unallocated space.

Why isn’t there a straightfowrward way to flash the external drive with required space size? Jetson ecosystem is not new, it should be mature enough by now.

KevinFFF · May 30, 2024, 6:43am

Could you use the following command instead?

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-only -c ./tools/kernel_flash/flash_l4t_nvme.xml --external-device nvme0n1 -S 100GiB --showlogs jetson-xavier-nx-devkit-emmc nvme0n1p1

Please also share the full flash log for further check.

neuroSparK · May 30, 2024, 7:14am

This command also failed to flash the NVme. Here is the log:
Jetpack5.1.2_NX_flash.log (24.7 KB)

KevinFFF · May 30, 2024, 4:35pm

First, are you using the custom BSP package from your vendor to flash the custom carrier board?

[   1.4711 ] End sector for APP, expected at: 122159070, actual: 0

It seems the fail is caused from the size issue.

Do you have any customization in partition layout file? (Linux_for_Tegra/tools/kernel_flash/flash_l4t_nvme.xml)
If so, please revert them and refer to the following configurations in your case.

physical size: 119.25GiB
In flash_l4t_nvme.xml: Please use num_sectors=230686720
Please use -S 100GiB in initrd flash command

neuroSparK · May 30, 2024, 7:19pm

Thanks a lot @KevinFFF you saved my 3 days of frustration. Setting num_sectors=230686720 worked finally. Btw, I wonder why is the num_sectors reported by the lsblk command wrong. I got the value by calculation too like bellow:

Number of sectors = 128,035,676,160 bytes / 512 bytes per sector = 250,069,680 sectors

Could you please tell how did you calculate the num_sectors ?

KevinFFF · May 31, 2024, 2:48am

230686720 (sectors)= 11010241024*1024 (bytes) / 512 (bytes/sector)

You can refer to the following thread for their relation.
How to solve the issue that ssd are not entiely availiable after full disk encryption - #5 by KevinFFF

system · June 14, 2024, 2:49am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.