I have the same problem with my (16 GB RAM) Xavier AGX. I upgraded it yesterday and it hung up in the boot, so I reflashed it today with the SDK manager. It comes up but if I do an update / upgrade it won’t boot any more. There must be something wrong with one of the packages on the upgrade. I’m going to wait a week before attempting another upgrade; I lost a day on my project for this and can’t afford to lose another day right now - I’m closing out a release.
I run the python to check modules then i got following error.
When i run the jetson-interface, also get a error massage.
All are works before update it.
seems like i can’t even install the SDK manager now haha, what a mess.
Hi,
Since there are many users in this thread, it means any kinds of issue could be here.
Thus, I suggest all people here please do share your serial console log to show and confirm whether we are really talking about the same issue.
For example, “stuck in NV boot logo” could represent either kernel error or bootloader error. We need to take these kinds of issue separately.
@wmindramalw, I am glad it works on your case. But your new issue seems not boot issue anymore.
Please file a new topic and tag me. We will check your new error there.
OK … but it’s going to be next week before I can do any bootloader troubleshooting. The machine’s been working for a month and a half without incident and along comes a 200+ package upgrade :-(
How to downgrade to the last version until fixing this?
This is best product i ever used. Thank you for this product to be invented. We are not getting big worry about this issue. Hope you solve this issue as soon as possible.
Another workaround if you see the bash (initrd) console in UART after running OTA upgrade.
1. reboot the device 7 times,
2. the device should be able to boot up properly.
3. edit the /boot/extlinux/extlinux.conf to do following changes:
- APPEND ${cbootargs} quiet
+ APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
4. Reboot the device, and run "dpkg-reconfigure nvidia-l4t-bootloader"
5. reboot device.
3 Likes
@WayneWWW I really wish I had read this 18 hours ago :)
FYI: It does seem the bad update is still available since I just had this happen an hour or so ago to my NX. Is it confirmed a reflash of the SD card will work? Where does the bootloader live exactly on the NX?
The sdcard image and sdkmanager should work. Only the OTA will crash because something wrong in bootloader update.
We are still trying to get this fix ASAP. Also, we shall already pulled off the functionality of OTA.
Thanks for the fix instructions, solving my mystery of why my AGX suddenly started working after multiple reboots, and the bootloader details. I didn’t even get a chance to apply the manual fix since it appears a new package does fix the boot command line arguments.
$ sudo apt update && sudo apt dist-upgrade
Get:1 file:/var/cuda-repo-10-2-local-10.2.89 InRelease
Ign:1 file:/var/cuda-repo-10-2-local-10.2.89 InRelease
Get:2 file:/var/visionworks-repo InRelease
Ign:2 file:/var/visionworks-repo InRelease
Get:3 file:/var/visionworks-sfm-repo InRelease
Ign:3 file:/var/visionworks-sfm-repo InRelease
Get:4 file:/var/visionworks-tracking-repo InRelease
Ign:4 file:/var/visionworks-tracking-repo InRelease
Get:5 file:/var/cuda-repo-10-2-local-10.2.89 Release [574 B]
Get:6 file:/var/visionworks-repo Release [2,001 B]
Get:7 file:/var/visionworks-sfm-repo Release [2,005 B]
Get:5 file:/var/cuda-repo-10-2-local-10.2.89 Release [574 B]
Get:8 file:/var/visionworks-tracking-repo Release [2,010 B]
Get:6 file:/var/visionworks-repo Release [2,001 B]
Get:7 file:/var/visionworks-sfm-repo Release [2,005 B]
Get:8 file:/var/visionworks-tracking-repo Release [2,010 B]
Hit:9 https://repo.download.nvidia.com/jetson/common r32.4 InRelease
Hit:10 https://repo.download.nvidia.com/jetson/t194 r32.4 InRelease
Hit:11 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:13 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Hit:14 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Hit:16 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
nvidia-l4t-bootloader nvidia-l4t-tools
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.6 MB of archives.
After this operation, 1,024 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://repo.download.nvidia.com/jetson/t194 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200709231533 [28.1 MB]
Get:2 https://repo.download.nvidia.com/jetson/t194 r32.4/main arm64 nvidia-l4t-tools arm64 32.4.3-20200709231533 [538 kB]
Fetched 28.6 MB in 3s (9,944 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 142544 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200709231533_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200709231533) over (32.4.3-20200625213407) ...
Preparing to unpack .../nvidia-l4t-tools_32.4.3-20200709231533_arm64.deb ...
Unpacking nvidia-l4t-tools (32.4.3-20200709231533) over (32.4.3-20200625213407) ...
Setting up nvidia-l4t-tools (32.4.3-20200709231533) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200709231533) ...
3668----1--jetson-xavier-nx-devkit-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.
Updating extlinux.conf...
Adding bootargs into exlinux.conf...
[username@hostname] -- [~]
$ cat /boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
# sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot
# LABEL backup
# MENU LABEL backup kernel
# LINUX /boot/Image.backup
# INITRD /boot/initrd
# APPEND ${cbootargs}
… and it reboots! Others might want to hold off updating until there’s more testing, however.
How is this issue going?
can we fix through update?
Hi,
New version should be posted already. Please try it.
I think it’s not working. At least in my configuration. I currently have it doing the swap to SSD hack from JetsonHacks. Not knowing of this issue, I ran a update/upgrade a couple hours ago and wasn’t able to reboot. So I did some web searching and came across this thread. Seeing that you’d literally just posted the update, I ran another apt update/upgrade and it did find a new bionic update (bionic-updates 4.15.0-111.112 arm64 [upgradable from: 4.15.0-109.110]). Rebooted and no love. After 7+ reboots (really I think it’s 10 for me), I copied boot/* and /lib/modules/4.9.140-tegra from the SSD to SD. Rebooted and STILL no love.
Anyway, I’m not 100% sure if the OTA update is still borked or if my hacked SD/SSD swap process has introduced another variable. But at least right now, I’m not able to boot without rebooting 10’ish times.
It is working fine on my Xavier AGX. Once the update to the bootloader was posted, I ran apt-get update && apt-get upgrade, rebooted and the system came up with no problems.
The whole rootfs. And sorry. That was 100% user error on my part. Apparently cp -R -f is NOT the same as rsync -a. :) It DOES boot when I copy the boot and module dirs correctly.
Any idea when we might be able to boot directly to SSD without that hack??
Thanks!
1 Like
Hi,
Let me briefly explain the issue.
- After rel-32.4.3, we put the root=boot_device in extlinux.conf but not in device tree anymore.
- However, the deb package forgets to update this. So your device is getting stuck in initrd bash console because it does not know where to load the rootfs.
Thus, our solution is to ask you to add root=/dev/mmcblk0p1 in extlinux.conf manually and device will be able to boot rootfs now.
However, in your case, you don’t put rootfs on /dev/mmcblk0p1 but on SSD, which I guess is something like /dev/sdb1…?
As a result, please try to put the correct device as root device.
Hmm. Interesting. The current hack basically installs a simple service that runs on boot which:
- mounts the SSD
- runs
systemctl switch-rootto the SSD
It requires that the SD still be mounted. I’m assuming that SD card would STILL be required in any case since it would still need to find the /boot folder there. But perhaps changing the root= line in extlinux.conf might avoid the need for the additional service to switch-root since it will mount the SSD as root during initrd?
Sorry. I know only enough to be dangerous…
Thanks!!
Do you happen to recall what your exact rsync commands were?
I’ve been working on this, of course the first thing to do is make a backup before trying anything so that’s taken a bit to put some scripts together.
The second part of the problem is a little tougher. You can update everything on the rootfs on the SSD with the update/upgrade, and then copy over the /boot directory. However, that doesn’t update all of the different libraries on the SD card (CUDA, CUDNN and so on). If someone feels that is important, to make it “correct” you would have to boot from the SD card and do the update/upgrade there too.
I’m thinking a better way is update/upgrade the SSD, then flash a new SD card (in this case with JetPack 4.4) and switch the root to the SSD in the /boot/exlinux.conf file on the SD card. That way everything should be ‘synced’ for the most part, but everything should be somewhat cleaner.
This gets rid of the idea of the service. The service was a workaround for 4.4DP being unable to set the rootfs in the extlinux.conf file. However, it is possible that modifying the extlinux.conf file can introduce errors that are difficult to debug without the use of the serial console (things like typos and such).
Just did:
sudo mkdir /mnt/sd/
sudo mount /dev/mmcblk0p1 /mnt/sd/
sudo rsync -a /boot/ /mnt/sd/boot/
sudo rsync -a /lib/modules/4.9.140-tegra/ /mnt/sd/lib/modules/4.9.140-tegra/
and rebooted. I THINK that worked. At the very least it didn’t break anything badly enough to prevent it from booting. :)
I think you’re probably right about the “better way.” It makes it so that you can remove the SSD and get mostly the same experience. Though this only solves the problem in the short term, no? Longer term as OTA updates continue, the SD will continue to fall out of sync. Ideally I’d think you’d want the SD to be nothing but a shell that maybe contains JUST the boot folder and nothing more. At least if the goal is to use the SSD as the “authoritative” root.
Again, though, I’m only thinking “logically,” not from any actual experience. And of course TRULY ideally there’d be a bootloader that would allow boot without the SD entirely. Though I do understand why this wouldn’t be a priority since the core NX module itself only has the SD card.
3 Likes