Flashing JetPack on an NVMe SSD on Xavier AGX

[bsbraven]

What I want
I would like to flash JetPack on my Samsung 970 Evo Plus SSD so that I can mount my root file system on it and have a lot more storage space for my ROS catkin workspace packages and associated bag files.

What I’ve tried
First, I flashed my SSD NVMe via SDKManager, which was successful. However, the APP partition was only 28 GiB and no matter what I tried, gparted would not allow me to transfer the rest of my SSD storage space (automatically sent to the UDA partition) to my APP partition, where my root file system was stored.

So, I used these instructions to flash my NVMe drive: https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E0QN0HA.

However, in step 10 of the instructions, it wants me to open the U-Boot prompt (or cboot for the AGX) to run the last command, run bootcmd_nvme0. When my Xavier is booting, no pressed key will trigger the cboot terminal though, so I am unable to run that command. (I’ve heard you can access cboot via the serial console terminal using gtkterm, but that won’t work either)

As a result, I can’t boot from my SSD (the SSD option comes up when I select ‘device to boot from’ when booting my Jetson, but it won’t allow me to select it.)

My Question
How do I flash / install my root file system on my SSD so that the full SSD storage space is utilized (i.e. all extra space is allocated to the APP partition?

Thank you!

[DaveYYY]

Hi,

What L4T version do you use?
Is there any reason you have to stick with rel-32?
Is it a DevKit or a custom board?

[bsbraven]

Hi Dave,

My L4T version is R35 Rev 4.1. No, there’s no reason I have to stick to R32, how come? And it’s the DevKit.

[DaveYYY]

Then you should not refer to documents in r32.
CBoot/UBoot is used in r32, whereas we use UEFI in r35.
Please try this:

JetPack 5.1.2/L4T 35.4.1 is now live

By further investigating, the root cause is that the layout used to flash external device is changed to “flash_l4t_t194_nvme.xml” from “flash_l4t_external.xml” in “nvsdkmanager_flash.sh” for T194 device. This only affects the customer who is using SDKM to flash device on T194 device. For customers who use initrd flash to flash device, there is no effect. For the customers who want to extend the size of APP partition on external device for T194 device, here is the workaround: Flashed the devi…

[bsbraven]

Oh okay great. As flashing is a time-intensive process I’d like to confirm:

1. I plug the NVMe into my Xavier AGX and then flash the NVMe via SDKManager

2. I find nvsdkmanager_flash.sh in the ~/nvidia/nvidia_sdk//Linux_for_Tegra/ directory (on my host computer?), and replace flash_l4t_t194_nvme.xml with flash_l4t_external.xml (where do I find the flash_l4t_external.xml file?)

3. Then I reflash my NVMe using SDJManager a second time?

Thank you

[bsbraven]

Update
I haven’t tried the above method yet, but I’ve tried the initrd method for R35 found here: Flashing Support — Jetson Linux<br/>Developer Guide 34.1 documentation

First, I’ve removed my SSD from my Jetson, plugged it into my host computer, ran

sudo BOARDID=2888 BOARDSKU=0004 FAB=400 ./tools/kernel_flash/l4t_initrd_flash.sh -c tools/kernel_flash/flash_l4t_external.xml -S 200GiB --external-device nvme0n1p1 --direct sdb jetson-agx-xavier-devkit external

on my host computer and of course edited the configuration file’s num_sectors size to allow my APP partition to take up 200 GiB of my SSD.

However, gparted indicated that the mount point of my APP partition is a directory in media instead of /, and when I press F11 when booting my Jetson to select which device to boot from in the UEFI GUI, my SSD does come up but it won’t let me boot it when I select it so I have to boot from eMMC. Is this because my mount point is not root?

Continuing the initrd instructions, I plug my Jetson into my host computer using the USB-C port designated for flashing, boot in recovery mode, and run sudo ./flash.sh jetson-agx-xavier-devkit nvme0n1p1 which gives me this message:

###############################################################################

L4T BSP Information:

R35 , REVISION: 4.1

User release: 0.0

###############################################################################
Error: probing the target board failed.
Make sure the target board is connected through
USB port and is in recovery mode.

I didn’t think it would be this difficult to give the Jetson a little extra storage space. What am I doing wrong? Thank you

[DaveYYY]

@bsbraven

• I plug the NVMe into my Xavier AGX and then flash the NVMe via SDKManager
• I find nvsdkmanager_flash.sh in the ~/nvidia/nvidia_sdk//Linux_for_Tegra/ directory (on my host computer?), and replace flash_l4t_t194_nvme.xml with flash_l4t_external.xml (where do I find the flash_l4t_external.xml file?)
• Then I reflash my NVMe using SDJManager a second time?

Yes, and you don’t need to find the file; just replace it inside the script.

[DaveYYY]

@bsbraven

Error: probing the target board failed.
Make sure the target board is connected through
USB port and is in recovery mode.

Are you sure it’s put into force recovery mode?

[bsbraven]

So it turns out my recovery button did not work. I physically connected the recovery button pins, booted it in recovery mode, and ran the last shell command and it worked perfectly. I will paste my process below in case anyone wants additional clarity on how to flash and boot from an SSD on an Nvidia Jetson (Xavier AGX in my case).

[bsbraven]

Booting the Jetson Xavier AGX from an SSD

A few notes

Though not a complicated process, this took me some time to figure out so this is simply a tutorial on how to configure an SSD to boot JetPack based on my experience flashing my Xavier AGX to boot from an SSD.

By default, the Jetson boots from its onboard storage (in the Xavier’s case, it’s a onboard 30 GiB storage device called its eMMC). I wanted it to boot from an NVMe - in this case the Samsung Evo 250 GiB SSD so that it had a lot more breathing room to store packages, bag files, etc. So, I used a SSD to USB adapter, plugged it into my host computer, and flashed it using these instructions. (The Flashing to an NVMe Drive section - for R35)

The command I used on step 4 of the instructions is

sudo BOARDID=2888 BOARDSKU=0004 FAB=400 ./tools/kernel_flash/l4t_initrd_flash.sh -c tools/kernel_flash/flash_l4t_external.xml -S 200GiB --external-device nvme0n1p1 --direct sdb jetson-agx-xavier-devkit external

If you want to personalize this command, use the link I provided above and the initrd README to do so.

Before running the above command though, I modified the second line of the flash_l4t_external.xml configuration file to

<device type="external" instance="0" sector_size="512" num_sectors="454863281"> <!-- num_sectors="NUM_SECTORS" -->

I’m not sure how necessary it was to modify the num_sectors size but that was my first approach and either it was indeed necessary, or it automatically corrected for it during the flash process.

Then, for the root file system to mount onto the NVMe, follow the 2 step process below the initial instructions in the above link, which is titled To set up an NVMe drive manually for using as root file system.

A summary of that process is to remove the NVMe from the host computer and install it on the Jetson (not via USB, but through the internal NVMe slot). Then, connect the Jetson to the host computer via USB cable, boot it in recovery mode, and run this command on your host computer’s shell:

cd ~/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra$

sudo ./flash.sh jetson-agx-xavier-devkit nvme0n1p1

Make sure the Jetson is booted in recovery mode before doing so.