Jetson Xavier Nx Not Booting

I run the python to check modules then i got following error.

When i run the jetson-interface, also get a error massage.

All are works before update it.

seems like i can’t even install the SDK manager now haha, what a mess.

Hi,

Since there are many users in this thread, it means any kinds of issue could be here.
Thus, I suggest all people here please do share your serial console log to show and confirm whether we are really talking about the same issue.

For example, “stuck in NV boot logo” could represent either kernel error or bootloader error. We need to take these kinds of issue separately.

@wmindramalw, I am glad it works on your case. But your new issue seems not boot issue anymore.

Please file a new topic and tag me. We will check your new error there.

OK … but it’s going to be next week before I can do any bootloader troubleshooting. The machine’s been working for a month and a half without incident and along comes a 200+ package upgrade :-(

How to downgrade to the last version until fixing this?

This is best product i ever used. Thank you for this product to be invented. We are not getting big worry about this issue. Hope you solve this issue as soon as possible.

Another workaround if you see the bash (initrd) console in UART after running OTA upgrade.

1. reboot the device 7 times,
2. the device should be able to boot up properly.
3. edit the /boot/extlinux/extlinux.conf to do following changes:
- APPEND ${cbootargs} quiet
+ APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
4. Reboot the device, and run "dpkg-reconfigure nvidia-l4t-bootloader"
5. reboot device.
2 Likes

@WayneWWW I really wish I had read this 18 hours ago :)

FYI: It does seem the bad update is still available since I just had this happen an hour or so ago to my NX. Is it confirmed a reflash of the SD card will work? Where does the bootloader live exactly on the NX?

The sdcard image and sdkmanager should work. Only the OTA will crash because something wrong in bootloader update.

We are still trying to get this fix ASAP. Also, we shall already pulled off the functionality of OTA.

@WayneWWW

Thanks for the fix instructions, solving my mystery of why my AGX suddenly started working after multiple reboots, and the bootloader details. I didn’t even get a chance to apply the manual fix since it appears a new package does fix the boot command line arguments.

 $ sudo apt update && sudo apt dist-upgrade 
Get:1 file:/var/cuda-repo-10-2-local-10.2.89  InRelease
Ign:1 file:/var/cuda-repo-10-2-local-10.2.89  InRelease
Get:2 file:/var/visionworks-repo  InRelease
Ign:2 file:/var/visionworks-repo  InRelease
Get:3 file:/var/visionworks-sfm-repo  InRelease
Ign:3 file:/var/visionworks-sfm-repo  InRelease
Get:4 file:/var/visionworks-tracking-repo  InRelease
Ign:4 file:/var/visionworks-tracking-repo  InRelease
Get:5 file:/var/cuda-repo-10-2-local-10.2.89  Release [574 B]
Get:6 file:/var/visionworks-repo  Release [2,001 B]                         
Get:7 file:/var/visionworks-sfm-repo  Release [2,005 B]                                                                
Get:5 file:/var/cuda-repo-10-2-local-10.2.89  Release [574 B]                                                          
Get:8 file:/var/visionworks-tracking-repo  Release [2,010 B]                                                         
Get:6 file:/var/visionworks-repo  Release [2,001 B]                                                                                                 
Get:7 file:/var/visionworks-sfm-repo  Release [2,005 B]                                                                                 
Get:8 file:/var/visionworks-tracking-repo  Release [2,010 B]                                                                                                     
Hit:9 https://repo.download.nvidia.com/jetson/common r32.4 InRelease                                                                                             
Hit:10 https://repo.download.nvidia.com/jetson/t194 r32.4 InRelease                                                                   
Hit:11 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:13 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Hit:14 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Hit:16 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Reading package lists... Done                      
Building dependency tree       
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  nvidia-l4t-bootloader nvidia-l4t-tools
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.6 MB of archives.
After this operation, 1,024 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://repo.download.nvidia.com/jetson/t194 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200709231533 [28.1 MB]
Get:2 https://repo.download.nvidia.com/jetson/t194 r32.4/main arm64 nvidia-l4t-tools arm64 32.4.3-20200709231533 [538 kB]
Fetched 28.6 MB in 3s (9,944 kB/s)        
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 142544 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200709231533_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200709231533) over (32.4.3-20200625213407) ...
Preparing to unpack .../nvidia-l4t-tools_32.4.3-20200709231533_arm64.deb ...
Unpacking nvidia-l4t-tools (32.4.3-20200709231533) over (32.4.3-20200625213407) ...
Setting up nvidia-l4t-tools (32.4.3-20200709231533) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200709231533) ...
3668----1--jetson-xavier-nx-devkit-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.
Updating extlinux.conf...
Adding bootargs into exlinux.conf...
[username@hostname] -- [~] 
 $ cat /boot/extlinux/extlinux.conf 
TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0

# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
#      sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot

# LABEL backup
#    MENU LABEL backup kernel
#    LINUX /boot/Image.backup
#    INITRD /boot/initrd
#    APPEND ${cbootargs}

… and it reboots! Others might want to hold off updating until there’s more testing, however.

How is this issue going?
can we fix through update?

Hi,

New version should be posted already. Please try it.

I think it’s not working. At least in my configuration. I currently have it doing the swap to SSD hack from JetsonHacks. Not knowing of this issue, I ran a update/upgrade a couple hours ago and wasn’t able to reboot. So I did some web searching and came across this thread. Seeing that you’d literally just posted the update, I ran another apt update/upgrade and it did find a new bionic update (bionic-updates 4.15.0-111.112 arm64 [upgradable from: 4.15.0-109.110]). Rebooted and no love. After 7+ reboots (really I think it’s 10 for me), I copied boot/* and /lib/modules/4.9.140-tegra from the SSD to SD. Rebooted and STILL no love.

Anyway, I’m not 100% sure if the OTA update is still borked or if my hacked SD/SSD swap process has introduced another variable. But at least right now, I’m not able to boot without rebooting 10’ish times.

Hi pault45z6,

Do you only put swap on SSD or even the whole rootfs is on SSD?

It is working fine on my Xavier AGX. Once the update to the bootloader was posted, I ran apt-get update && apt-get upgrade, rebooted and the system came up with no problems.

The whole rootfs. And sorry. That was 100% user error on my part. Apparently cp -R -f is NOT the same as rsync -a. :) It DOES boot when I copy the boot and module dirs correctly.

Any idea when we might be able to boot directly to SSD without that hack??

Thanks!

1 Like

Hi,

Let me briefly explain the issue.

  1. After rel-32.4.3, we put the root=boot_device in extlinux.conf but not in device tree anymore.
  2. However, the deb package forgets to update this. So your device is getting stuck in initrd bash console because it does not know where to load the rootfs.

Thus, our solution is to ask you to add root=/dev/mmcblk0p1 in extlinux.conf manually and device will be able to boot rootfs now.

However, in your case, you don’t put rootfs on /dev/mmcblk0p1 but on SSD, which I guess is something like /dev/sdb1…?

As a result, please try to put the correct device as root device.

Hmm. Interesting. The current hack basically installs a simple service that runs on boot which:

  1. mounts the SSD
  2. runs systemctl switch-root to the SSD

It requires that the SD still be mounted. I’m assuming that SD card would STILL be required in any case since it would still need to find the /boot folder there. But perhaps changing the root= line in extlinux.conf might avoid the need for the additional service to switch-root since it will mount the SSD as root during initrd?

Sorry. I know only enough to be dangerous…

Thanks!!

Do you happen to recall what your exact rsync commands were?
I’ve been working on this, of course the first thing to do is make a backup before trying anything so that’s taken a bit to put some scripts together.

The second part of the problem is a little tougher. You can update everything on the rootfs on the SSD with the update/upgrade, and then copy over the /boot directory. However, that doesn’t update all of the different libraries on the SD card (CUDA, CUDNN and so on). If someone feels that is important, to make it “correct” you would have to boot from the SD card and do the update/upgrade there too.

I’m thinking a better way is update/upgrade the SSD, then flash a new SD card (in this case with JetPack 4.4) and switch the root to the SSD in the /boot/exlinux.conf file on the SD card. That way everything should be ‘synced’ for the most part, but everything should be somewhat cleaner.

This gets rid of the idea of the service. The service was a workaround for 4.4DP being unable to set the rootfs in the extlinux.conf file. However, it is possible that modifying the extlinux.conf file can introduce errors that are difficult to debug without the use of the serial console (things like typos and such).

Just did:

sudo mkdir /mnt/sd/
sudo mount /dev/mmcblk0p1 /mnt/sd/
sudo rsync -a /boot/ /mnt/sd/boot/
sudo rsync -a /lib/modules/4.9.140-tegra/ /mnt/sd/lib/modules/4.9.140-tegra/

and rebooted. I THINK that worked. At the very least it didn’t break anything badly enough to prevent it from booting. :)

I think you’re probably right about the “better way.” It makes it so that you can remove the SSD and get mostly the same experience. Though this only solves the problem in the short term, no? Longer term as OTA updates continue, the SD will continue to fall out of sync. Ideally I’d think you’d want the SD to be nothing but a shell that maybe contains JUST the boot folder and nothing more. At least if the goal is to use the SSD as the “authoritative” root.

Again, though, I’m only thinking “logically,” not from any actual experience. And of course TRULY ideally there’d be a bootloader that would allow boot without the SD entirely. Though I do understand why this wouldn’t be a priority since the core NX module itself only has the SD card.

3 Likes

Thank you for the commands. If it boots, more than likely things are OK. Note that the update/upgrade itself may have broken things here and there because of broken dependencies. That’s to be expected, nature of the beast.

Those are good thoughts. Other than the boot function, the SD card is superflous. There’s some comfort in having a full system on the SD card, but probably isn’t that big of deal either way. You can always generate a new one in a bind, or have an extra.

I know that it was a lot of work changing the boot loaders, which eventually swapped UBoot for CBoot. Bootloaders are tricky, availability of different drivers during the boot sequence and such. I wouldn’t think CBoot has access to NVMe drivers. However, I think it is a good feature request to be able to boot from SSD on the Xaviers.

2 Likes