I accidently missed a tilda in command and lost the whole system.
I was trying to flash the nvme using the sdkmanager, but it fails with error “failed to read rcm_state”.
I have Jetson AGX Orin 64GB
My host is Ubuntu 24 LTS
Can somebody please advice for how to restore?
Hi,
Please refer to the below topics.
Thanks
Thanks for your reply.
Here is the lsusb :
Bus 001 Device 061: ID 0955:7023 NVIDIA Corp. APX
I tried the login:
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --initrd jetson-agx-orin-devkit internal
The login fails: Log is attached.
How do I go to device manager? Is it through some tool or device?
pksingh pksingh System.txt (113.0 KB)
My Host is as follows:
I suggest you clone the rootfs if you need the content, and then reflash. You could use cloned content to restore some parts, such as home directory content; you’d have to manually fix things like directory permissions, but it would become possible to do so while leaving system files fresh from flash (and thus the o/s itself would work correctly).
I don’t need the content. I just need the device to work again, even if from a fresh start.
You would need to flash again, but instructions differ depending on type. I think an Ubuntu 24 host might work when flashing L4T R36.x (L4T is just what you call the flashed content when the NVIDIA drivers are added to it; JetPack/SDK Manager is the GUI front end to the flash software). However, if this is a dev kit with an SD card on the module itself, and no eMMC, then you only need a clone of one of the prebuilt SD card images; if this is an eMMC model, then you’ll need that host PC to flash.
FYI, the dev kits which don’t have the eMMC put the software equivalent of a BIOS, plus boot content, in QSPI memory located on the module itself. For dev kits which have eMMC, this content goes in the eMMC partitions. The major release going into the o/s (in the rootfs partition) must be compatible with the boot content. For example an L4T R36.x rootfs works with R36.x boot content, but not with R35.x boot content. Changing the permissions only changes content in the rootfs (o/s), and so in theory you don’t need the rest of the content. However, when building partitions for the eMMC models this is difficult, and so you’d just flash everything for that model. For an SD card model the QSPI is likely still working, and so for that model, if you just put a new prebuilt image onto an SD card, then it would boot up and allow that first boot account setup without actual flash.
Here is a list of L4T releases:
https://developer.nvidia.com/linux-tegra
(this also leads to the prebuilt SD card images and JetPack/SDKM if flashing)
Thanks for the nice explanation, really appreciate it.
Is the APX an issue?
I got this device from Amazon. Amazon.com: NVIDIA Jetson AGX Orin 64GB Developer Kit : Electronics
Are there any set of commands to copy the image created to device?
Hi,
Please try
sudo -s
echo -1 > /sys/module/usbcore/parameters/autosuspend
and redo the commands
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --initrd jetson-agx-orin-devkit internal
Also some questions to confirm
Thanks
Thanks a lot. It is native machine running only OS of Ubuntu. I posted the machine details above.
I get the same error.
[ 0.3132 ] BL: version 1.4.0.4-t234-54845784-e89ea9bc last_boot_error: 0
[ 0.3823 ] Sending bct_mem
[ 0.4075 ] Sending blob
[ 0.5096 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --instance 1-1.2 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
Cleaning up…
The content of this file is as hee:
-1
�������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
APX usually shows up under virtualization. In and of itself, this does not cause problems, but any virtualized environment tends to be passing along USB from another environment. If that passing is not “permanent”, then as USB disconnects and reconnects there is a tendency to lose USB. The flash software has no knowledge and no ability to configure a virtual environment. Also, there are often times when a virtual environment has underestimated required disk space (and perhaps used the wrong filesystem type). Just watch for USB issues or disk space issues.
I tried with another Ubuntu laptop It shows the same APX there too.
For the power source I have connected with USB-C which is above the DC port. And the USB-C with host is cnnected to the other USB C port. Is that correct ?
I only got a USB to USB-C wire and a USB C power adapter. Just guessing if power is the issue.
I fear that device is bricked.
Any of the power methods work. FYI, it is very difficult to “brick” a Jetson. Bricking usually means the BIOS is corrupt, and that the chip used for BIOS has to be unsoldered and put into a programmer to fix it, then soldered back in. Jetsons don’t have a hardware BIOS, they just have software. When you do a complete flash, not only are flashing o/s and bootloader, you are also flashing the equivalent of everything BIOS.
The APX in and of itself is not an error. The only time it matters is when using a VM. A lot of people come here asking for help and don’t mention the VM, but really it is the setup of the VM causing issues. The APX won’t matter if you’re not actually using a container or VM, but it is still a curiosity.
What seems odd is that when you “cat /sys/module/usbcore/parameters/autosuspend” it should show only the “-1”, and nothing after that. Was it a copy and paste issue, or was there a lot of “nonsense” echoed back when you “cat /sys/module/usbcore/parameters/autosuspend”? Incidentally, if you want this to stop autosuspend, and if there is a VM, then both the host o/s and the VM would need to have autosuspend disabled.
Is there any kind of error when you do what @DavidDDD suggested?
sudo echo -1 > /sys/module/usbcore/parameters/autosuspend
(the file being echoed to is not really a file; it is an interface to the USB driver)
Whenever you do that, then you can check if there was success:
cat /sys/module/usbcore/parameters/autosuspend
Previous day there was a lot of ascii garbage on screen after cat. So I posted all that. Today it is only -1.
There is no error with @DavidDDD 's commands.
I tried with flashcmd. That fails too. I notice that the device USB gets disconnected(the device disappears from lsusb) right before the install.
root@pksingh-System-Product-Name:/media/pksingh/SD1/nvidia/nvidia_sdk/JetPack_6.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader# bash ./flashcmd.txt
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commandsEntering RCM boot
[ 0.0215 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from --mb1_bin
[ 0.0215 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from --psc_bl1_bin
[ 0.0215 ] rcm boot with presigned binaries
[ 0.0223 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[ 0.0232 ] BR_CID: 0x80012344705DF180800000000F0201C0
[ 0.1216 ] Sending bct_br
[ 0.1219 ] Sending mb1
[ 0.1231 ] Sending psc_bl1
[ 0.1319 ] Sending bct_mb1
[ 0.1377 ] Generating blob for T23x
[ 0.1413 ] tegrahost_v2 --chip 0x23 0 --generateblob blob.xml blob.bin
[ 0.1421 ] The number of images in blob is 19
[ 0.1438 ] blobsize is 81801222
[ 0.1440 ] Added binary blob_uefi_jetson_minimal_with_dtb_sigheader.bin.encrypt of size 2158656
[ 0.2026 ] Added binary blob_pscfw_t234_prod_sigheader.bin.encrypt of size 310768
[ 0.2033 ] Added binary blob_mce_flash_o10_cr_prod_sigheader.bin.encrypt of size 187120
[ 0.2042 ] Added binary blob_tsec_t234_sigheader.bin.encrypt of size 176128
[ 0.2050 ] Added binary blob_applet_t234_sigheader.bin.encrypt of size 279808
[ 0.2057 ] Not supported type: mb2_applet
[ 0.2060 ] Added binary blob_mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt of size 440944
[ 0.2064 ] Added binary blob_xusb_t234_prod_sigheader.bin.encrypt of size 164864
[ 0.2067 ] Added binary blob_nvpva_020_sigheader.fw.encrypt of size 2164640
[ 0.2070 ] Added binary blob_display-t234-dce_sigheader.bin.encrypt of size 12075728
[ 0.2125 ] Added binary blob_nvdec_t234_prod_sigheader.fw.encrypt of size 294912
[ 0.2162 ] Added binary blob_bpmp_t234-TE990M-A1_prod_sigheader.bin.encrypt of size 1027008
[ 0.2180 ] Added binary blob_tegra234-bpmp-3701-0005-3737-0000_with_odm_sigheader.dtb.encrypt of size 265024
[ 0.2185 ] Added binary blob_camera-rtcpu-t234-rce_sigheader.img.encrypt of size 458096
[ 0.2193 ] Added binary blob_adsp-fw_sigheader.bin.encrypt of size 415008
[ 0.2197 ] Added binary blob_spe_t234_sigheader.bin.encrypt of size 270336
[ 0.2201 ] Added binary blob_tos-optee_t234_sigheader.img.encrypt of size 1887312
[ 0.2229 ] Added binary blob_eks_t234_sigheader.img.encrypt of size 9232
[ 0.2232 ] Added binary blob_boot.img of size 58959872
[ 0.2681 ] Added binary blob_tegra234-p3737-0000+p3701-0005-nv.dtb of size 254662
[ 0.3241 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
[ 0.3250 ] BL: version 1.4.0.4-t234-54845784-e89ea9bc last_boot_error: 0
[ 0.4055 ] Sending bct_mem
[ 0.4057 ] Sending blob
[ 0.5413 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
root@pksingh-System-Product-Name:/media/pksingh/SD1/nvidia/nvidia_sdk/JetPack_6.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader#
dmseglog.txt (261.3 KB)
I am also attaching the DMSEG log, if this gives.
Short answer: There is indeed a USB failure. The log with the disconnect/reconnect and reenumerating with a new device number over and over is a purely USB issue.
When the Jetson is in recovery mode it becomes a custom USB device. JetPack/SDK Manager is just a front end to the actual flash software (appropriately known as the “driver package” since it is a custom USB device). Once flash is complete the Jetson will automatically reboot. If flash completed, then it is ok for USB to disappear, at least for a short time.
As the Jetson boots up normally (including after a flash automatic reboot), then the software in the Jetson itself provides a virtual wired Ethernet connection over the USB. This is when USB comes back in normal operation because the Jetson looks like an Ethernet router due to the software. The optional software installs, e.g., CUDA, are then installed over ssh to the admin account you would have set up on first boot.
If for some reason normal reboot occurs, but the virtual wired Ethernet does not come up (which can be due to an unsuccessful flash), then there might have been a flash failure.
The place where USB can disappear during the actual flash, before rebooting, is normal. The Jetson will in fact disconnect and reconnect during the flash. If any kind of VM is involved, then usually the USB fails to reconnect due to VM settings (the Jetson has no control over that).
If for some reason the Jetson disappears during normal flash stages, but it is not due to a VM, then sometimes the USB cable itself is an issue. I’ve not seen this to be an issue on the Orin due to the USB-C though. The older models which use a micro-OTG connector are the ones where the cable itself is suspect. You might try a different port if this is not due to reboot.
In the log I see the APX listed. Normally I would see this only on a VM. You’ve not answered if the host is a native host, or if it is a VM. All indications are that your VM needs to be told to own that USB device, such that if the device unplugs, then when it replugs the VM still owns it.
I have mentioned before also. The host in Native Ubuntu 24 LTS host. I have tried with an Ubuntu laptop too. It is always APX.
If this is native, then I don’t think the APX would matter. However, as an experiment, does the host PC’s BIOS have a way to disable virtualization? If so, then turn this off in BIOS and try again. See if the APX disappears, and also if USB is not lost.
Should that fail, then when it gets to the failure point, try unplugging the USB and then plugging it back in.
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --initrd --network eth0:OrinIP/24:Ubuntu24.04IP:RouterIP \
jetson-agx-orin-devkit internal
Linux_for_Tegra/tools/kernel_flash/l4t_initrd_flash.func still contains the following. I’ve not tried with Jetpack 6.2 but it worked for me several times to use eth0 IP when my agx orin 32gb would not flash because usb0 kept disapearing.
–network Flash through Ethernet protocal using initrd flash. can be "usb0" to flash through the USB Flashing cable
or "eth0:/:[:]" to flash through the LAN cable
For examples:
–network usb0
–network eth0:192.168.0.17/24:192.168.0.21
–network eth0:192.168.0.17/24:192.168.1.2:192.168.0.1
Here’s what I have historically used
./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1
-c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/generic/cfg/flash_t234_qspi.xml”
–showlogs --network eth0:192.168.1.79/24:192.168.1.179:192.168.1.1
jetson-agx-orin-devkit internal
Normally network would not matter during actual flash, and would only matter after flash completes (at which time the Jetson automatically reboots and is no longer in recovery mode; this is when optional software is added, and flash is actually complete and no longer in recovery mode). However, if you are using some sort of network flash whereby an image comes over NFS, then you would need networking during the flash. The wired Ethernet would likely be the most reliable way to do this, but then you might need to be certain the fully booted Jetson (remember, the recovery mode Jetson automatically reboots without telling you it has done so) still has everything it needs for networking. I don’t know the details of your flash commands though, so I’m not qualified to answer beyond this.
I’ll suggest this is where @DavidDDD might be able to comment on those initrd flash commands using the network. It sounds though like maybe flash actually succeeded and automatically rebooted, and then something fails. Maybe.