Flashing to NVMe on new Jetson Orin Nano Super (36.4.4) failing

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

I have a brand-new JONS with firmware 36.4.4, and it boots fine from an SD card imaged with JetPack 6.2.1, but when I try to follow the instructions at 🛸 Initial Setup (SDK Manager method) - NVIDIA Jetson AI Lab to use the SDK Manager method to install to the NVMe SSD (a 2TB Samsung 2280), it consistently fails with the following errors:

08:06:14.118 - Info: /home/wjmaclean/nvidia/nvidia_sdk/JetPack_6.2.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra
08:06:14.119 - Info: ***************************************
08:06:14.119 - Info: *                                     *
08:06:14.119 - Info: *  Step 3: Start the flashing process *
08:06:14.119 - Info: *                                     *
08:06:14.119 - Info: ***************************************
08:06:14.250 - Info: Waiting for target to boot-up...

…

08:08:36.166 - Info: Waiting for target to boot-up...
08:08:37.167 - Info: Timeout
08:08:37.168 - Info: Device failed to boot to the initrd flash kernel. Please retrive the serial log during flashing to debug further.
08:08:37.168 - Info: Cleaning up...
08:08:37.274 - Error: [exec_command]: /bin/bash -c /home/wjmaclean/.nvsdkm/replays/scripts/JetPack_6.2.1_Linux/NV_L4T_FLASH_JETSON_LINUX_COMP.sh; [error]: rpcbind: another rpcbind is already running. Aborting
08:08:37.274 - Info: [ Component Install Finished with Error ]08:08:37.274 - Info: [host] [ 16.02 GB used. Disk Avail on Partition /dev/mapper/vgubuntu-root: 248.13 GB ]08:08:37.274 - Info: [ NV_L4T_FLASH_JETSON_LINUX_COMP Install took 7m46s ]

So it seems to want me to “retrieve the serial log during flashing to debug further”, but I’m not quit sure what that means. I’m guessing a USB-TTL serial cable?

I formatted the SSD to ext4 when I booted from the SD card.

All the installation steps work fine until the errors at the end of the flashing step. Any suggestions? I’ve ordered a USB-TTL serial converter so I can observe what’s happening.

2

Hello @w.james.maclean,

Hope everything is going great!

Yeah you’re right, retrive the serial log during flashing means using a USB-TTL cable to read UART output while the flashing procedure is undergoing.

One debug step you could take is to try to flash manually without the SDK manager.

You just need to go into your Linux_for_tegra directory inside the ~/nvidia directory where SDK manager downloads everything.

Then you can flash the device with the following command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
  -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" \
  --showlogs --network usb0 jetson-orin-nano-devkit-super internal

Please let us know how it goes.

best regards,
Andrew
Embedded Software Engineer at ProventusNova

3

I’m afraid that didn’t help. I’ve attached the log for this command (flash_log02.txt below).

I got a USB-TTL cable, and the failure when doing the SDK Manager approach seems to occur after the following output on the debug console:

insmod /lib/modules/5.15.148-tegra/updates/drivers/net/ethernet/nvidia/nvethernet/nvethernet.ko ^M^M
insmod /lib/modules/5.15.148-tegra/kernel/drivers/usb/typec/stusb160x.ko ^M^M   
insmod /lib/modules/5.15.148-tegra/updates/drivers/platform/tegra/mce/tegra-mce.ko ^M^M
insmod /lib/modules/5.15.148-tegra/updates/drivers/spi/spi-tegra210-quad.ko ^M^M 
[    6.934386] tegra-qspi 3270000.spi: Adding to iommu group 9^M                
[    6.934992] tegra-qspi 3270000.spi: Prod config not found for QSPI: -19^M    
[    6.936074] spi-nor spi0.0: mx25u51279g (65536 Kbytes)^M                     
[    7.954625] using random self ethernet address^M                             
[    7.954629] using random host ethernet address^M                             
[    7.982331] usb0: HOST MAC 22:cd:59:fa:ac:6e^M                               
[    7.982336] usb0: MAC 62:97:57:48:18:6d^M                                    
[    7.983465] tegra-xudc 3550000.usb: EP 0 (type: ctrl, dir: out) enabled^M    
bash: cannot set terminal process group (-1): Inappropriate ioctl for device^M^M
bash: no job control in this shell^M^M                                          
bash-5.1#

(for full output see log01.txt, attached
log01.txt (91.4 KB))

I’ve seen this error elsewhere (e.g. here and here), but neither of these issues seems to have had a helpful resolution. The first link suggests that the issue is that the OS is “not able to find your rootfs (file system) on the pre-set location”. Since the whole point of this setup is to put the root fs on the nvme SSD drive, it seems like that hasn’t worked somehow.

The debug output shows the bash-5.1 prompt while the SDK Manager is waiting for the initrd boot to finish, so I’m guessing that whatever script on the JONS that failed hadn’t sent the boot confirmation to the host yet. It seems like the next thing to do is figure out exactly what is failing in the JONS flash script so it can be fixed.

flash_log02.txt (285.5 KB)

4

Hi,

Please help answer the questions first. Are you using a NV devkit or a custom board? or you are not sure what does I mean here? (if you really don’t know, please just tell that you don’t know).

And whether your host PC supports RNDIS or not?

The current log just indicates the USB device mode on Jetson is not detected by your host.

BTW, since you are using Orin Nano, please do not refer to any forum post in “Jetson Nano” forum. Orin Nano and Nano are totally two different things. Orin is new and Jetson nano is a product that released 6 years ago.

5

I am using a (very recent) brand new Jetson Orin Nano Super dev kit.

The host PC is Ubuntu 22.04 LTS (running on a Lenovo Carbon X1 5th Gen). I’m guessing this supports RNDIS, but if there’s a test I can do to confirm please let me know.

How can I determine what the host (running SDK Manager) thinks the Jetson’s USB Device mode, and what is the correct device mode I should be looking for? When I started the flashing process, the SDK Manager detected the board saying “SDK Manager has detected a connected device which is running in recovery mode.” … is that indicative of the USB device mode? Or am I looking for something else.

Ok … I had just hoped that since the error messages were the same, that these other posts might shed some light into what’s wrong … even if the underlying boards are different, presumably there is some overlap in the support software, no?

6

When your UART console got stuck in this stage,

[ 7.982331] usb0: HOST MAC 22:cd:59:fa:ac:6e
[ 7.982336] usb0: MAC 62:97:57:48:18:6d
[ 7.983465] tegra-xudc 3550000.usb: EP 0 (type: ctrl, dir: out) enabled
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.1#

Please hotplug your type C usb flash cable. And check the “dmesg” on your host and also the uart console on your Jetson and see if you see any new log.

When Jetson is detected as usb device, lsusb command on your host PC should show that.

When I started the flashing process, the SDK Manager detected the board saying “SDK Manager has detected a connected device which is running in recovery mode.”

The flash process of a Orin Nano would be like “first, recovery mode” and then “usb device mode after running some init process with initrd”. So it is not sufficient that sdkmanager told you your board is in recovery mode.

Ok … I had just hoped that since the error messages were the same, that these other posts might shed some light into what’s wrong … even if the underlying boards are different, presumably there is some overlap in the support software, no?

Nope, the software on Jetson Nano and Orin Nano has a 6-year gap. Really have nothing much left there to refer to a post from Jetson nano for Orin Nano issue.

7

Just to confirm before I try this: you mean unplug and then reinsert the USB-C cable? The USB-C cable is plugged in at the start of the flashing process.

8

Ok, I tried what you suggested, and it helped a little, but the flashing still fails.

The initial USB info from the host (prior to starting flashing with SDK Manager) is:

dmesg:

[2678547.621717] usb 7-2: new high-speed USB device number 2 using xhci_hcd
[2678547.749333] usb 7-2: New USB device found, idVendor=0955, idProduct=7523, bcdDevice= 4.01
[2678547.749353] usb 7-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[2678547.749362] usb 7-2: Product: APX
[2678547.749369] usb 7-2: Manufacturer: NVIDIA Corp.

lsusb:
Bus 007 Device 002: ID 0955:7523 NVIDIA Corp. APX

Here’s the UART Console Log:
log03.txt (92.9 KB)

After the USB disconnect/reconnect, it seems the console output resumed for a bit, but there is an RNDIS init failure in the DMESG output. On the host side we have:

dmesg:

[2679137.001954] usb 7-2: new high-speed USB device number 2 using xhci_hcd
[2679137.134611] usb 7-2: New USB device found, idVendor=0955, idProduct=7035, bcdDevice= 0.01
[2679137.134635] usb 7-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2679137.134648] usb 7-2: Product: Linux for Tegra
[2679137.134658] usb 7-2: Manufacturer: NVIDIA
[2679137.134667] usb 7-2: SerialNumber: 1421625088909
[2679137.569966] rndis_host 7-2:1.0: RNDIS init failed, -110
[2679137.570007] rndis_host: probe of 7-2:1.0 failed with error -110
[2679137.571822] usb 7-2: USB disconnect, device number 2

lsusb:

Bus 007 Device 003: ID 0955:7035 NVIDIA Corp. Linux for Tegra

The failure message in the SDK Manager terminal is:

14:06:45.534 - Info: /home/wjmaclean/nvidia/nvidia_sdk/JetPack_6.2.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra
14:06:45.535 - Info: ***************************************
14:06:45.535 - Info: *                                     *
14:06:45.535 - Info: *  Step 3: Start the flashing process *
14:06:45.535 - Info: *                                     *
14:06:45.535 - Info: ***************************************
14:06:45.758 - Info: Waiting for target to boot-up...
     <MANY OF THESE LINES>
14:09:07.409 - Info: Waiting for target to boot-up...
14:09:08.411 - Info: Timeout
14:09:08.411 - Info: Device failed to boot to the initrd flash kernel. Please retrive the serial log during flashing to debug further.
14:09:08.413 - Info: Cleaning up...
14:09:08.495 - Error: [exec_command]: /bin/bash -c /home/wjmaclean/.nvsdkm/replays/scripts/JetPack_6.2.1_Linux/NV_L4T_FLASH_JETSON_LINUX_COMP.sh; [error]: rpcbind: another rpcbind is already running. Aborting
14:09:08.495 - Info: [ Component Install Finished with Error ]14:09:08.495 - Info: [host] [ 4.75 MB used. Disk Avail on Partition /dev/mapper/vgubuntu-root: 226.01 GB ]14:09:08.495 - Info: [ NV_L4T_FLASH_JETSON_LINUX_COMP Install took 7m48s ]

There are a bunch of RNDIS related messages at the end of the debug console log, like:

[   18.173807] rndis_msg_parser: unknown RNDIS message 0x0052033A len 4456526
[   18.173809] RNDIS command error -524, 0/24
[   18.176113] tegra-xudc 3550000.usb: sequence number error
[   18.176118] rndis_msg_parser: unknown RNDIS message 0x0052033A len 4456526
[   18.176119] RNDIS command error -524, 0/24
10

Do you have other host PC that can try?

Looks like RNDIS fails to launch.

11

Not unless I can use a MacBook as the host.

Otherwise, it would be easier to try and fix RNDIS on my Ubuntu machine … can I try a different USB-C port? Maybe unplug other USB devices on that machine?

12

Another option: since I can boot to the SD card fine, is there a way to convert the installation to boot from the SDD entirely on the JONS itself? Then RNDIS wouldn’t be an issue. At present the SSD is formatted, but otherwise empty.

13

So I managed to borrow another Ubuntu 22.04 machine, and the flashing did indeed work ok.

But just as a feature request, it would be nice if it was possible to flash from the device, after booting from an SD card.