Native boot from M.2 NVME SSD

No just

Sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1

What Kangalow means is the initrd flash command use internally within the script nvsdkmanager_flash.sh

1 Like

gotcha; so it seems that finally empty NX devkit with attached nvme ssd could be flashed with just one single comand execution, right?

This repository has a more detailed set of scripts: GitHub - jetsonhacks/bootFromExternalStorage: Shell scripts to setup a NVIDIA Jetson AGX Xavier or Jetson Xavier NX Developer Kit to boot from external storage.

You need to install the Secure Boot package in addition to the BSP and sample rootfs on the host, then flash the Jetson. There are also some dependencies that need to be installed on the host to support the installation. After flashing, there is a script to run on the Jetson which installs the NVIDIA JetPack packages. This better matches the default SD Card experience on the Xavier NX.

For some reason, I have not been able to get the nvsdkmanager_flash.sh script to work with my USB SSD yet, but that’s a different topic.

1 Like

Yes

For usb nvme, if you plug it through the usb port. You can try

Sudo ./nvsdkmanager_flash.sh --storage sda1

nvme0n1p1 is for nvme ssd plugged directly in the m.2 slot

Yes, that is the change I made. The USB SSD (a Samsung T5, 250GB) is ‘flashed’ as expected. I’ve tried this two different ways, booting with the drive attached directly to the Xavier NX, and attached through a powered USB 3.0 hub. However, when the Jetson boots:

# Attached to Xavier
[0004.082] I> ########## USB (0) boot ##########
[0004.087] W> No valid slot number is found in scratch register
[0004.087] W> Return default slot: _a
[0004.101] I> Validate XUSB-FW ...
[0004.101] I> T19x: Authenticate XUSB-FW (bin_type: 11), max size 0x30000
[0004.103] I> Encryption fuse is not ON
[0004.103] I> USB Firmware Version: 60.09 release
[0004.161] I> regulator of usb2-0 already enabled
[0004.165] I> regulator of usb2-1 already enabled
[0004.170] I> regulator of usb2-2 already enabled
[0005.236] I> USB 2.0 port 2 new high-speed USB device detected
[0005.237] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0005.338] I> Start to enumerate device
[0005.340] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0005.349] I> 
[0005.350] I> Enumerated USB Device 0bda:5489
[0005.350] I> 
[0005.351] I> Hub:
[0005.351] I>   Downstream ports        : 4
[0005.352] I>   Power switching         : Ganged
[0005.352] I>   Compound device         : No
[0005.352] I>   Over-current protection : Global
[0005.353] I>   Hub avg delay           : 4 ns
[0005.355] I> Set interface = 1 (alt setting)
[0005.361] I> Enable hub ports
[0005.573] I> Device detected on port 2
[0005.575] I> Device detected on port 4
[0006.182] I> Device detected on port 3
[0006.384] I> Reset port 2
[0006.439] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0006.540] I> Start to enumerate device
[0006.542] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0006.545] I> This device is non-MSD, skip enumeration
[0006.548] I> Reset port 3
[0006.603] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad11000
[0006.704] I> Start to enumerate device
[0006.705] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad11000
[0006.709] I> This device is non-MSD, skip enumeration
[0006.712] I> Reset port 4
[0006.766] I> Port 4 device is low speed, skip enumeration
[0006.766] E> Enumerated device doesn't belong to MSD class or protocol is not !
[0006.767] E> Class = 9, Protocol = 1
[0006.767] E> USBMSD: Failed to initialize Enumerated USB device either not an .
[0006.772] E> Error opening USBMSD driver 0, err: 7c7c0312
[0006.777] W> Failed to initialize device 5-0
[0006.781] E> USB boot failed, err: 2088502034
[0006.786] I> ########## NVME (0) boot ##########







# Attached to hub
[0004.082] I> ########## USB (0) boot ##########
[0004.087] W> No valid slot number is found in scratch register
[0004.087] W> Return default slot: _a
[0004.100] I> Validate XUSB-FW ...
[0004.101] I> T19x: Authenticate XUSB-FW (bin_type: 11), max size 0x30000
[0004.102] I> Encryption fuse is not ON
[0004.103] I> USB Firmware Version: 60.09 release
[0004.160] I> regulator of usb2-0 already enabled
[0004.165] I> regulator of usb2-1 already enabled
[0004.169] I> regulator of usb2-2 already enabled
[0005.235] I> USB 2.0 port 2 new high-speed USB device detected
[0005.237] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0005.338] I> Start to enumerate device
[0005.339] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0005.349] I> 
[0005.349] I> Enumerated USB Device 0bda:5489
[0005.349] I> 
[0005.351] I> Hub:
[0005.352] I>   Downstream ports        : 4
[0005.352] I>   Power switching         : Ganged
[0005.352] I>   Compound device         : No
[0005.353] I>   Over-current protection : Global
[0005.353] I>   Hub avg delay           : 4 ns
[0005.355] I> Set interface = 1 (alt setting)
[0005.360] I> Enable hub ports
[0005.571] I> Device detected on port 1
[0005.574] I> Device detected on port 4
[0006.181] I> Device detected on port 3
[0006.383] I> Reset port 1
[0006.439] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0006.539] I> Start to enumerate device
[0006.541] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad10000
[0006.545] I> This device is non-MSD, skip enumeration
[0006.547] I> Reset port 3
[0006.602] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad11000
[0006.703] I> Start to enumerate device
[0006.705] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa4ad11000
[0006.708] I> This device is non-MSD, skip enumeration
[0006.711] I> Reset port 4
[0006.765] I> Port 4 device is low speed, skip enumeration
[0006.766] E> Enumerated device doesn't belong to MSD class or protocol is not !
[0006.766] E> Class = 9, Protocol = 1
[0006.766] E> USBMSD: Failed to initialize Enumerated USB device either not an .
[0006.771] E> Error opening USBMSD driver 0, err: 7c7c0312
[0006.776] W> Failed to initialize device 5-0
[0006.781] E> USB boot failed, err: 2088502034
[0006.785] I> ########## NVME (0) boot ##########

a short time later the Jetson becomes very unhappy since it cannot find a boot device. It appears that it does not see the drive during enumeration, or does not identify it as a Mass Storage Device (MSD).

I see. USB boot is tested with USB flash drive plugged directly in the Jetson. So USB NVMe or USB hub might not work properly.

That makes sense. I’ll try that and see if it works.

Just so that I can understand. What is the objection that a lot of people have to having the SD card used for just a fraction of a single during the boot process? I understand this in the context of making boxes where you want to make it harder for anyone to change the boot process or pull the card out, such as giving a box to a non-technical customer. But other than that I’m not seeing where the problem is. Surely not the cost of the SD card?

But not worrying about the sd card remaining in the system the complexity can be reduced by the approach I suggested with GitHub - hcfman/sbts-boot-from-SSD: One command installation of boot from SSD for Jetson nano, Xavier NX and Xavier AGX

I’d like to understand the usecase/psychology a bit better.

Kim Hendrikse

Maybe my previous comment was not clear enough.

For those who want to have a easy entrance to jetson, they can use the sdcard and sdcard image.
And this is the default method we suggest to user because we don’t know how familiar and how much efforts that each user want to put to learn this boot process.

If you are familiar enough with all the boot process, flash process on jetson, then in fact you don’t need sdcard, you can use other boot device to replace it since it is only rootfs + kernel.

However, the software support for each boot device is different in each software release. For example, for jetson NX, jetpack releases < 4.6.1 don’t support NVMe boot at all.

Thus, we have different workarounds in each software release on this forum and this may make you feel confused.

Okay, thanks. I’m bringing out a project soon but I don’t support custom approaches to configuring the setup of the boot process of the project itself. Because the project doesn’t need any customer kernels it can do that. It’s a variation of the approach I mention above, so I just wanted to understand where people’s concerns were. Thanks for the answer.

Kim

Does anyone have an idea why flashed NX devkit after applying the script GitHub - jetsonhacks/bootFromExternalStorage: Shell scripts to setup a NVIDIA Jetson AGX Xavier or Jetson Xavier NX Developer Kit to boot from external storage. won’t run further than displaying the Nvidia logo? Neither it will be shown on lsusb outputs unless put into the recovery mode.
In the outputs I can see successful execution but for this

Waiting for target to boot-up...
Waiting for device to expose ssh .........Run command: if [ -f /qspi/l4t_flash_from_kernel.sh ]; then USER=root /qspi/l4t_flash_from_kernel.sh --no-reboot --qspi-only ; fi on root@fe80::1%enp0s20u1
blockdev: cannot open /dev/sdd: No medium found
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/bootFromExternalStorage/R32.6.1/Linux_for_Tegra/tools/kernel_flash/images/internal/flash.idx
Number of lines is 48
max_index=47
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
Flash index file is /qspi/internal/flash.idx
Number of lines is 48
max_index=47
[ 0]: l4t_flash_from_kernel: Starting to flash to qspi
Erasing 64 Kibyte @ 0 --  0 % complete [ 2]: l4t_flash_from_kernel: Successfully create gpt for emmc
[ 2]: l4t_flash_from_kernel: Starting to create gpt for external device
Active index file is /mnt/bootFromExternalStorage/R32.6.1/Linux_for_Tegra/tools/kernel_flash/images/external/flash.idx
Erasing 64 Kibyte @ 10000 --  0 % complete Number of lines is 14
max_index=13

flash_3-1_0_20210829-074812.log (62.8 KB)
The problem is that after trying the initrd thing now sdkmanager flashing also fails on the new nx devkit unit given sdcard is attached:

│info: exec_command: /tmp/tmp_NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_CO │
│   Retry                                                             ││MP.sh                                                                │
│   Exit                                                              ││info: sudo ./nvautoflash.sh                                          │
│                                                                     ││info: *** Checking ONLINE mode ... OK.                               │
│                                                                     ││info: *** Checking target board connection ... 1 connections found.  │
│                                                                     ││info: *** Reading ECID ... *** Error: ECID read failed.              │
│                                                                     ││info: Put the target board in RCM mode and retry.                    │
│                                                                     ││info: [ Component Install Finished with Error ]

but lsusb shows the board in the recovery mode; trying again after restarting the devkit ; the failure is below:

o: [  29.5284 ] [...] 100%                                        │
│                                                                     ││info:                                                                │
│                                                                     ││info: [  29.6408 ] Erasing sdcard: 0 ... 0000000074740311: E>        │
│                                                                     ││NV3P_SERVER: Failed to open instance 0 of device 6.                  │
│                                                                     ││info: [  29.6433 ]                                                   │
│                                                                     ││info: [  29.6433 ] [Failed]                                          │
│                                                                     ││info: [  29.6434 ]                                                   │
│                                                                     ││info: Error: Return value 17

However, next attempt also fails with

info: [ NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP Install took 4m5s  │
│                                                                     ││]                                                                    │
│                                                                     ││error: [error]: : [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FLASH │
│                                                                     ││_XAVIER_NX_WITH_OS_IMAGE_COMP.sh; [error]: 20+0 records in           │
│                                                                     ││info: 20+0 records out                                               │
│                                                                     ││info: 20 bytes copied, 0.000511135 s, 39.1 kB/s                      │
│                                                                     ││info:                                          

so the only outcome is that the unit got kind of non useable after trying the initrd thing. It took all Sunday to do 10+ flashing attempts of which every one failed. too bad: losses - 1 day of life, nx devkit unit usability; gains - none.

After trying to flash the Jetson with the JetsonHacks method, did you have the SD card in the slot?

Hi Jim. Thank you for following up.
First I removed the sdcard so only samsung nvme ssd remained. then I tried the script from github flash_jetson_external_storage.sh. Then I tried the flashing without the script from the folder downloaded by the script using Sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1. Then I tried Sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1 from folder created by sdkmanager. Then when they all failed I added sdcard then tried regular sdkmanager flashing; Finally I unscrewed nvme ssd then tried few attempts with sdkmanager flashing.
Each of steps above tried 3-5 times. I did not try flashing with version other than 4.6 or with sdcard image. So the result seems that the unit won’t flash or boot

@Andrey1984 Did you remove the force recovery jumper before the flash?

@Kangalow I do not have the jumper, I just shortcut the FRC/GND pins with metal screwdriver once while powering on
Now I will have to try flashing with JP 4.5, 4.4, from sdcard image that will take more time
JP4.5 failed

info: [  28.7369 ] [...] 100%                                        │
│   Retry                                                             ││info:                                                                │
│   Exit                                                              ││info: [  28.8493 ] Erasing sdcard: 0 ... 0000000074740311: E>        │
│                                                                     ││NV3P_SERVER: Failed to open instance 0 of device 6.                  │
│                                                                     ││info: [  28.8521 ]                                                   │
│                                                                     ││info: [  28.8521 ] [Failed]                                          │
│                                                                     ││info: [  28.8521 ]                                                   │
│                                                                     ││info: Error: Return value 17                                         │
│                                                                     ││info: Command tegradevflash_v2 --pt flash.xml.bin --create           │
│                                                                     ││info: Failed flashing t186ref.                                       │
│                                                                     ││info: *** ERROR: flashing failed.

another attempt

17:42:47 INFO: Verified, 0955:7e19 detected in recovery mode.
17:42:47 DEBUG: running command < true >
17:42:47 INFO: command finished successfully
17:42:47 DEBUG: running command < /tmp/device_mode_host_setup.sh >
17:42:47 INFO: Found route default wlp2s0, setting up device mode
17:42:47 INFO: net.ipv4.ip_forward = 1
17:42:47 INFO: Device mode on host successfully set with DNS 8.8.8.8!
17:42:47 INFO: command finished successfully
17:42:47 DEBUG: running command < true >
17:42:47 INFO: command finished successfully
17:42:47 DEBUG: Flash Jetson Xavier NX - flash: using adapter to install NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP@JETSON_XAVIER_NX_TARGETS to /mnt/installation_nx/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS
17:42:48 INFO: Flash Jetson Xavier NX - flash: Installing component 'Flash Jetson Xavier NX' (NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP)
17:42:48 INFO: Flash Jetson Xavier NX - flash: sudo ./nvautoflash.sh
17:42:48 INFO: Flash Jetson Xavier NX - flash: *** Checking ONLINE mode ... OK.
17:42:48 INFO: Flash Jetson Xavier NX - flash: *** Checking target board connection ... 1 connections found.
17:42:53 INFO: Flash Jetson Xavier NX - flash: *** Reading ECID ... *** Error: ECID read failed.
17:42:53 ERROR: Flash Jetson Xavier NX - flash: command terminated with error
17:42:53 SUMMARY: DateTime Target Setup - target: Depends on failed component

How did you prepare the Linux_for_Tegra folder?
Before you did the nvsdkmanager_flash.sh did you turn off the udisks2 service?

Initially I tried just the github script
Then I tried sdkmanager. For sdkmanager variant I had to manually download the secureboot package. With the sdkmanager generated Linux_for_Tegra I just unpacked the secureboot, but did not do more specific steps like enabling or disabling something [However, I will try once again despite existing
time constraints]. It seems by now I am able to flash to sdcard from sdkmanager already[seems sdcard was not completely in during previous trials].
So this time I removed the sdcard, then unpacked secureboot then executed

sudo systemctl stop udisks2.service

trying again with

 nvsdkmanager_flash.sh

The result is that it is flashing successfully [as per the output], but it doesn’t show on boot anything but the nvidia Boot Logo; The logfile attached[sdcard detached]
flash_3-1_0_20210830-055438.log (63.6 KB)
the nvme ssd I am using - samsung 980pro; previously I used it with sdcard with JetsonHacks script bootfromssd
why does it populate mmcblk0p1 in the excerpt below?

*  Step 2: Generate rcm boot commandline *
*                                        *
******************************************
/mnt/installation_nx/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/flash.sh  --no-flash --rcm-boot jetson-xavier-nx-devkit-qspi mmcblk0p1
###############################################################################
# L4T BSP Information:
# R32 , REVISION: 6.1

however it gets further like

00 in flash
[ 137]: l4t_flash_from_kernel: Successfully flash the qspi
[ 137]: l4t_flash_from_kernel: Flashing success
tar: Read checkpoint 430000
writing item=13, 9:0:secondary_gpt, 34359721472, 16896, gpt_secondary_9_0.bin, 16896, fixed-<reserved>-0, 565d329e98028835e4f672b000180238e3426ccf
[ 181]: l4t_flash_from_kernel: Successfully flash the external device
[ 181]: l4t_flash_from_kernel: Flashing success
[ 181]: l4t_flash_from_kernel: The device size indicated in the partition layout xml is smaller than the actual size. This utility will try to fix the GPT.

Note: The flash process might have added some nvidia-flash-* connections in
NetworkManager to set up USB ethernet through flashing port (L4t USB Device mode).
You might want to check your NetworkManager configuration if you have some special
configuration.
Reboot target
Run command: sync; { sleep 1; reboot; } >/dev/null & on root@fe80::1%enp0s20u1
Success
Cleaning up...
Log is saved to Linux_for_Tegra/initrdlog/flash_3-1_0_20210830-062434.log 

but then it reboots then ever appears in lsusb, unless put into the recovery mode; no sdcard attached

Interesting, this log is pretty much the same as the one here that works. The next step is to provide a serial console output of the Jetson when it is booting to see where it gets stuck.

I have only the micro-usb to usb-A cable to connect jetson NX to the Host PC, but it [lsusb]won’t see the connection. Without extra cables I am not sure which logs might be acquired. I have though separate wires [ remnants of jetbot] for gPIO that I could wire from NX to AGX Jetson GPIO probably