Flashing Nano Devkit NVMe over USB fails

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

Hey,

I am trying to flash the Nano Devkit with Jetpack 5.1.1

I am using the following command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml --no-systemimg"  jetson-orin-nano-devkit-nvme external

I also tried

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi_nvme.xml --no-systemimg" jetson-orin-nano-devkit-nvme external

The flash process flashes the QSPI and then tells me that flashing the non qspi failed.
I have devices /dev/sde /dev/sdf /dev/sdg and /dev/sdh on my host when flashing fails. The flash script tells me the devide sdf does not exist. Which is correct:

fdisk: cannot open /dev/sde: No medium found


fdisk: cannot open /dev/sdf: No medium found


fdisk: cannot open /dev/sdg: No medium found


Disk /dev/sdh: 232.91 GiB, 250059350016 bytes, 488397168 sectors
Disk model: 0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

The actual device is the /dev/sdh, but flashing seems to attempt to flash /dev/sdf?

Here is the log:
flash.log (22.8 KB)

I tried the very same command with the Xavier NX Devkit, there is it working:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml --no-systemimg" jetson-xavier-nx-devkit external

And I get this output:
flashworking.log (28.0 KB)

Any idea what could be wrong? From the logs I’d think that it does not even attempt to flash the nvme…

Ignore the following part, it was possible to get the board back working with flash.sh

Unfortunately after trying to flash Xavier I attempted Nano again and get the following error:

[   0.1097 ] BL: version 0.32.0.0-t234-54845784-57325615 last_boot_error: 0
[   0.2813 ] Sending bct_mem
[   0.2817 ] Sending blob
[   0.3550 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --instance 1-1 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
Cleaning up...

??
[0004.896] I> MB1 (version: 0.32.0.0-t234-54845784-57325615)
[0004.902] I> t234-A01-1-Silicon (0x12347) Prod
[0004.906] I> Boot-mode : Coldboot
[0004.909] I> Emulation:
[0004.911] I> Entry timestamp: 0x00000000
[0004.915] I> last_boot_error: 0x0
[0004.918] I> BR-BCT: preprod_dev_sign: 0
[0004.922] I> rst_source: 0x0, rst_level: 0x0
[0004.926] I> Task: Bootchain select WAR set (0x5000ba65)
[0004.931] I> Task: Enable SLCG (0x5000bab1)
[0004.935] I> Task: CRC check (0x5001ea19)
[0004.939] I> Task: Initialize MB2 params (0x5000cb51)
[0004.945] I> MB2-params @ 0x40060000
[0004.948] I> Task: Crypto init (0x5001d981)
[0004.952] I> Task: Secure debug controls (0x5000c0a9)
[0004.957] I> Task: strap war set (0x5000ba2d)
[0004.961] I> Task: Initialize SOC Therm (0x5001bd35)
[0004.966] I> Task: Program NV master stream id (0x5000c05d)
[0004.972] I> Task: Verify boot mode (0xd4820f1)
[0004.978] I> Task: Alias fuses (0x5001095d)
[0004.982] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0004.989] I> Task: Print SKU type (0x5000f5f1)
[0004.993] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x000001c8
[0004.998] I> FUSE_OPT_GPC_DISABLE = 0x00000002
[0005.003] I> FUSE_OPT_TPC_DISABLE = 0x000000f0
[0005.007] I> FUSE_OPT_DLA_DISABLE = 0x00000003
[0005.011] I> FUSE_OPT_PVA_DISABLE = 0x00000001
[0005.016] I> FUSE_OPT_NVENC_DISABLE = 0x00000001
[0005.020] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0005.024] I> FUSE_OPT_FSI_DISABLE = 0x00000001
[0005.029] I> FUSE_OPT_EMC_DISABLE = 0x0000000c
[0005.033] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0005.037] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0005.041] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0005.045] I> FUSE_SKU_INFO_0 = 0xd5
[0005.049] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0005.053] I> FUSE_PACKAGE_INFO_0 = 0x2
[0005.056] I> SKU: Prod
[0005.059] I> Task: Boost clocks (0x500148a1)
[0005.063] I> Initializing PLLC2 for AXI_CBB.
[0005.067] I> AXI_CBB : src = 35, divisor = 0
[0005.071] I> Task: Voltage monitor (0x50014b49)
[0005.075] I> VMON: Vmon re-calibration and fine tuning done
[0005.081] I> Task: UPHY init (0x5000d065)
[0005.087] I> HSIO UPHY init done
[0005.090] E> Skipping GBE UPHY config
[0005.093] I> Task: Boot device init (0x50000be9)
[0005.098] I> Boot_device: RCM
[0005.101] I> USB configuration success
[0005.104] I> Task: TSC init (0x50020a4d)
[0005.108] I> Task: Load membct (0x50011fe9)
[0005.113] I> RAM_CODE 0x4000021
[0005.116] I> Loading MEMBCT
[0005.118] I> Slot: 0
[0005.120] I> Binary[0] block-0 (partition size: 0x40000)
[0005.125] I>  get_binary_info: Binary name: MEM-BCT-0
[0005.130] I> Size of crypto header is 8192
[0005.134] I> BCH load address is : 0x40050000
[0005.138] I> Size of crypto header is 8192
[0005.143] I> BCH of MEM-BCT-0 read from storage
[0005.147] I> BCH address is : 0x40050000
[0005.151] I> MEM-BCT-0 header integrity check is success
[0005.156] I> Binary magic in BCH component 0 is MEM0
[0005.161] I> component binary type is 0
[0005.168] I> MEM-BCT-0 binary is read from storage
[0005.173] I> MEM-BCT-0 binary integrity check is success
[0005.178] I> Binary MEM-BCT-0 loaded successfully at 0x40040000 (0xe580)
[0005.185] I> RAM_CODE 0x4000021
[0005.190] I> RAM_CODE 0x4000021
[0005.194] I> Task: Load Page retirement list (0x500115b1)
[0005.199] I> Task: SDRAM params override (0x50011fc5)
[0005.204] I> Task: Save mem-bct info (0x50014fa1)
[0005.209] I> Task: Carveout allocate (0x50015005)
[0005.213] I> ECC region[0]: Start:0x0, End:0x0
[0005.218] I> ECC region[1]: Start:0x0, End:0x0
[0005.222] I> ECC region[2]: Start:0x0, End:0x0
[0005.226] I> ECC region[3]: Start:0x0, End:0x0
[0005.230] I> ECC region[4]: Start:0x0, End:0x0
[0005.235] I> Non-ECC region[0]: Start:0x80000000, End:0x80000000
[0005.240] I> Non-ECC region[1]: Start:0x0, End:0x0
[0005.245] I> Non-ECC region[2]: Start:0x0, End:0x0
[0005.250] I> Non-ECC region[3]: Start:0x0, End:0x0
[0005.254] I> Non-ECC region[4]: Start:0x0, End:0x0
[0005.264] E> BL_CARVEOUT: Failed to allocate memory of size 0x8000000 for CO:31.
[0005.272] C> Task 0x0 failed (err: 0x49490003)
[0005.276] E> Top caller module: BL_CARVEOUT, error module: BL_CARVEOUT, reason: 0x03, aux_info: 0x00
[0005.285] C> Boot Info Table status dump :
1
[0005.289] I> Busy Spin

I rebooted everything, did I somehow brick the module?

3

Hi, may I ask how exactly is your device connected to your PC?
You are to flash the NVMe SSD drive installed in your Jetson module, what does it have to do with disks on your host PC?

Did you follow the flashing command provided in our developer guide?
https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/IN/QuickStart.html#jetson-modules-and-configurations

4

Your comment pointed me to the right direction.

I was used to the --network command in combination with eth0. I did not notice that the manual wrote “usb0” and omitted that part.

Is there any reason the flash for Orin uses NFS while the flash for Xavier uses mounted RNDIS USB Mass Storage devices?

Anyway after successfully flashing the board with this command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml --no-systemimg" --network usb0 --no-flash jetson-orin-nano-devkit internal

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only

it does not boot:

[0000.444] E> BL_CARVEOUT: Failed to allocate memory of size 0x8000000 for CO:31.
[0000.452] C> Task 0x0 failed (err: 0x49490003)
[0000.456] E> Top caller module: BL_CARVEOUT, error module: BL_CARVEOUT, reason: 0x03, aux_info: 0x00
[0000.465] C> Boot Info Table status dump :
1
[0000.469] I> Busy Spin
5

Hi, so now you successfully flashed your device, but it failed to boot?
Can you post the full log dumped from serial connection?

6

Yeah, the issue was the missing --network usb0

The flash process compared to xavier seems to use an NFS share now instead of mounting only the disks as RNDIS USB Mass storage devices.

The log is quite short. MB1 seems to fail:

??
[0000.063] I> MB1 (version: 0.32.0.0-t234-54845784-57325615)
[0000.068] I> t234-A01-1-Silicon (0x12347) Prod
[0000.073] I> Boot-mode : Coldboot
[0000.076] I> Emulation:
[0000.078] I> Entry timestamp: 0x00000000
[0000.082] I> last_boot_error: 0x49490003
[0000.086] I> BR-BCT: preprod_dev_sign: 0
[0000.089] I> rst_source: 0x3, rst_level: 0x1
[0000.094] I> Task: Bootchain select WAR set (0x5000ba65)
[0000.099] I> Task: Enable SLCG (0x5000bab1)
[0000.103] I> Task: CRC check (0x5001ea19)
[0000.107] I> Task: Initialize MB2 params (0x5000cb51)
[0000.112] I> MB2-params @ 0x40060000
[0000.116] I> Task: Crypto init (0x5001d981)
[0000.120] I> Task: Secure debug controls (0x5000c0a9)
[0000.125] I> Task: strap war set (0x5000ba2d)
[0000.129] I> Task: Initialize SOC Therm (0x5001bd35)
[0000.134] I> Task: Program NV master stream id (0x5000c05d)
[0000.139] I> Task: Verify boot mode (0xd4820f1)
[0000.145] I> Task: Alias fuses (0x5001095d)
[0000.150] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0000.157] I> Task: Print SKU type (0x5000f5f1)
[0000.161] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x000001c8
[0000.166] I> FUSE_OPT_GPC_DISABLE = 0x00000002
[0000.171] I> FUSE_OPT_TPC_DISABLE = 0x000000f0
[0000.175] I> FUSE_OPT_DLA_DISABLE = 0x00000003
[0000.179] I> FUSE_OPT_PVA_DISABLE = 0x00000001
[0000.183] I> FUSE_OPT_NVENC_DISABLE = 0x00000001
[0000.188] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0000.192] I> FUSE_OPT_FSI_DISABLE = 0x00000001
[0000.196] I> FUSE_OPT_EMC_DISABLE = 0x0000000c
[0000.201] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0000.205] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0000.209] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0000.213] I> FUSE_SKU_INFO_0 = 0xd5
[0000.216] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0000.221] I> FUSE_PACKAGE_INFO_0 = 0x2
[0000.224] I> SKU: Prod
[0000.226] I> Task: Boost clocks (0x500148a1)
[0000.230] I> Initializing PLLC2 for AXI_CBB.
[0000.235] I> AXI_CBB : src = 35, divisor = 0
[0000.239] I> Task: Voltage monitor (0x50014b49)
[0000.243] I> VMON: Vmon re-calibration and fine tuning done
[0000.249] I> Task: UPHY init (0x5000d065)
[0000.255] I> HSIO UPHY init done
[0000.258] E> Skipping GBE UPHY config
[0000.261] I> Task: Boot device init (0x50000be9)
[0000.266] I> Boot_device: QSPI_FLASH instance: 0
[0000.270] I> Qspi clock source : pllc_out0
[0000.274] I> QSPI Flash: Macronix 64MB
[0000.278] I> QSPI-0l initialized successfully
[0000.282] I> Task: TSC init (0x50020a4d)
[0000.286] I> Task: Load membct (0x50011fe9)
[0000.290] I> RAM_CODE 0x4000421
[0000.293] I> Loading MEMBCT
[0000.296] I> Slot: 1
[0000.298] I> Binary[0] block-0 (partition size: 0x40000)
[0000.303] I>  get_binary_info: Binary name: MEM-BCT-0
[0000.308] I> Size of crypto header is 8192
[0000.312] I> BCH load address is : 0x40050000
[0000.316] I> Size of crypto header is 8192
[0000.320] I> BCH of MEM-BCT-0 read from storage
[0000.325] I> BCH address is : 0x40050000
[0000.329] I> MEM-BCT-0 header integrity check is success
[0000.334] I> Binary magic in BCH component 0 is MEM0
[0000.338] I> component binary type is 0
[0000.343] I> MEM-BCT-0 binary is read from storage
[0000.348] I> MEM-BCT-0 binary integrity check is success
[0000.353] I> Binary MEM-BCT-0 loaded successfully at 0x40040000 (0xe580)
[0000.359] I> RAM_CODE 0x4000421
[0000.365] I> RAM_CODE 0x4000421
[0000.369] I> Task: Load Page retirement list (0x500115b1)
[0000.374] I> Task: SDRAM params override (0x50011fc5)
[0000.379] I> Task: Save mem-bct info (0x50014fa1)
[0000.384] I> Task: Carveout allocate (0x50015005)
[0000.388] I> RCM blob carveout will not be allocated
[0000.393] I> ECC region[0]: Start:0x0, End:0x0
[0000.397] I> ECC region[1]: Start:0x0, End:0x0
[0000.401] I> ECC region[2]: Start:0x0, End:0x0
[0000.406] I> ECC region[3]: Start:0x0, End:0x0
[0000.410] I> ECC region[4]: Start:0x0, End:0x0
[0000.414] I> Non-ECC region[0]: Start:0x80000000, End:0x80000000
[0000.420] I> Non-ECC region[1]: Start:0x0, End:0x0
[0000.425] I> Non-ECC region[2]: Start:0x0, End:0x0
[0000.429] I> Non-ECC region[3]: Start:0x0, End:0x0
[0000.434] I> Non-ECC region[4]: Start:0x0, End:0x0
[0000.444] E> BL_CARVEOUT: Failed to allocate memory of size 0x8000000 for CO:31.
[0000.451] C> Task 0x0 failed (err: 0x49490003)
[0000.455] E> Top caller module: BL_CARVEOUT, error module: BL_CARVEOUT, reason: 0x03, aux_info: 0x00
[0000.464] C> Boot Info Table status dump :
1
[0000.468] I> Busy Spin
7

Little update from my side. I was surprised that the guide now uses this command addition instead of relying on the EMMC_CFG in the conf file of the board:

 -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml"

So I created my own board config with the board EMMC_CFG I want set:
jetson-orin-nx-devkit-custom-nvme.conf

source "${LDK_DIR}/p3768-0000+p3767-0000.conf";

EMMC_CFG="flash_t234_qspi.xml";

Then I omitted the whole -p command which is used to set the config.

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml  --network usb0 jetson-orin-nano-devkit-custom-nvme internal

This works…and the board boots! I copied the whole flash command from this thread initially:

The only remaining difference I can spot is the “–no-systemimg” command.

So I tried the following command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh  --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --network usb0 jetson-orin-nano-devkit internal

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only

This still does not work despite it SHOULD do the same as mine?!

This command does work:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml  --network usb0 jetson-orin-nano-devkit-custom-nvme internal

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only

What is the reason behind the whole “-p” command? And any idea why it does not work for me?

8

Hi

Sorry to jump in.

Could we get a brief summary of what did you and hit issue? Also, is it devkit or custom board?

9

Hi @WayneWWW
We received our Jetson Orin Nano Devkit yesterday. So I went ahead and wanted to flash it.
Inititally I was trying the flash command from here:

That command is the same as in the developer guide.

I initially used it wrong because I omitted the “–network usb0” part. It was working like that on Xavier NX, so I read “–network eth0” and assumed I won’t need it.

Obviously the way of flashing was changed between Xavier and Orin. Inititally you used to have an RNDIS gadget on the device which mounted it’s NVMe partition on the host so the host could write to the device.
The new method seems to use an NFS server on the host exporting parts of the L4T directory so that the device can access it directly and use the data to flash itself. I am not sure if I am happy about the NFS share yet, but that’s another topic.

After fixing the command so that it looks exactly the same as in the dev guide I managed to flash the board but it is not able to boot.
I moved the

-p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml"

part into a .conf file as the EMMC_CFG and the otherwise exactly same command flashes correctly and the system is able to boot.

I was using custom .conf files to represent all the possible combinations between devkits/modules and our custom board. Due to the change in the flash commands that has become a bit cumbersome as I do not only have to change the .conf file but also adapt the flash command to the selected board in my flash script…I would’t really need the conf files if I need to parse what I flash anyway.

A suggestion from my side:
The “-c [NAME].xml” was added to the flash commands for NVMe flashing.
How about adding an “EXTERNAL_CFG” variable to the conf files for parsing by initrd_flash.sh so that the user can write the parameters for NVMe flashing in there? You need to switch the EMMC_CFG from qspi_sd or qspi_emmc to qspi anyway, so a new config file is required for NVMe flashing.

11

Hi @seeky15,
Finally which command is working to flash in MVMe and boot from same ?

12

Hey @varada.k, the solution has not been verified by NVIDIA but I am using this successfully:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_nvme.xml  --network usb0 jetson-orin-nano-devkit-custom-nvme internal

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only

Important ist that you need to create your own config for the device.
What’s important there is that you add an EMMC_CFG which only flashes the qspi.

Please use a config like this and place it under your L4T top directory:

source "${LDK_DIR}/p3768-0000+p3767-0000.conf";
EMMC_CFG="flash_t234_qspi.xml";

Use these configs in your command:
A/B Support: tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml
Single Filesystem: tools/kernel_flash/flash_l4t_nvme.xml

I need to unplug the board and reconnect it between the first and the second command because it becomes unreachable to the flash script after the first command.

Still waiting for @WayneWWW or @DaveYYY answer why the command in the developer guide does not work.

13

Hi seeky15,

Actually I am not sure where you got that command from. So I don’t know what you want to ask here.

Our developer guide is directly using one step flash command but not --no-flash + --flash-only.

14

Hey @WayneWWW

that’s from the tools/kernel_flash/README_initrd_flash.txt

Anyway, the two step process is not what makes it fail. It does not work with one or two steps.
The --no-flash command is used so that you can create the flash files without having the actual board attached. The --flash-only can then reuse that data multiple times without always recreating the data.

As you can see in the very first comment I also tried it without the “flash” options

15

Hi seeky15,

Honestly, I have to say that you always put your story too long and actually I don’t have time to read your whole story. Sorry that none of us is native Englisher speaker.

Could you just share it as a item list

  1. What is your current error/issue?

  2. What is the diff between your setup and default config from BSP?

16

I am sorry, you asked for the summary. I thought I’d give you a complete summary so that you could completely understand the issue.

  1. Quick Start — Jetson Linux Developer Guide documentation

To Flash the Jetson Developer Kit Operating Software

Point 6, first Command
→ does not work
The flash succeeds but the bootloader is not working

[0000.444] E> BL_CARVEOUT: Failed to allocate memory of size 0x8000000 for CO:31.
[0000.452] C> Task 0x0 failed (err: 0x49490003)
[0000.456] E> Top caller module: BL_CARVEOUT, error module: BL_CARVEOUT, reason: 0x03, aux_info: 0x00
[0000.465] C> Boot Info Table status dump :
1
[0000.469] I> Busy Spin

  1. Jetson Orin Nano devkit with Jetson Orin devkit edition module and NMVe
    Rootfs and kernel is customized. But the bootloader or uefi has not been touched. I’d say standard setup.

  2. The old commands we know from xavier series still work.

17

Are you talking about Jetson Orin Nano devkit + Jetson Orin devkit edition module +NMVe cannot work with below command even with default BSP?

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1
-c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml”
–showlogs --network usb0 jetson-orin-nano-devkit internal

18

Yeah, that’s how it seems to me. I’ve only read about people in the forum using the same command, but they already fail with a timeout.

Maybe I am doing something wrong, missing a character or so. But for me this command does not work.
I have not done, and I will not do the test with unmodified rootfs. I’ve done that numerous times to 90% find out that the issue was not on my side. The chance that my rootfs has an influence on MB1 should be zero.

I’ve found an undocumented working command now. If you want, ignore my posts.
But maybe you want to see if it actually works for others or not.

19

Hi,

Could you share the full uart logs of below two cases

  1. The original command

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1
-c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml”
–showlogs --network usb0 jetson-orin-nano-devkit internal

  1. The command that you ran and it could flash successfully.
20

And also the info of your module after you boot it up.

cat /etc/nv_boot_control.conf

21

Can do that next week tuesday. I don’t have the Orin board here at the moment. But I can do that.

1 Like
22

Ok, please let me check the log and see what is going on.