Recovery Mode

Hello,

I accidentally put a program that hangs in the /etc/rc.local file with the exit 0 return afterwards. Because of this my Jetson wont boot into ubuntu and I can’t edit the file. I’ve tried holding the regular Ubuntu keys during the bios load but they didn’t work. Whats the best way to get shell access to the box without loading the /etc/rc.local file?

Ben

Embedded systems don’t have a BIOS. Any halt of boot would be via U-Boot, but Xavier skips U-Boot and goes directly from CBoot to Linux.

Serial console is the definitive way to get in when all else fails. Although it is possible this file could interfere even with serial console it is unlikely to stop this.

On the same side of the Xavier where the 40-pin header is, note the type-C USB on the right, and a micro-USB on the left as you face the header. This micro-USB, when it has a micro-B type cable inserted (and this is what NVIDIA provides) to a non-recovery mode Xavier, offers some “devices” if the other end is connected to your PC. I don’t know what this will be identified on for your PC, but if you monitor “dmesg --follow” as you connect this it should tell you. On one of my systems this shows up as “/dev/serial/by-id/usb-FTDI_Quad_RS232-HS-if03-port0”.

Any serial console program should work, but I like gtkterm (“sudo apt-get install gtkterm”). The proper settings are speed 115200, 8 bits, no parity, one stop bit. Hardware flow control is not used. For gtkterm this would be:

sudo gtkterm -b 8 -t 1 -s 115200 -p /dev/serial/by-id/usb-FTDI_Quad_RS232-HS-if03-port0

Note that “sudo” is only required if your PC user is not a member of the “dialup” group. An admin user is probably already a member of this and won’t need sudo. Try without sudo first, and if you get a permission denied, only then try with sudo.

This console will show boot messages, and after boot will offer a text mode login. The user logged in as will be the admin user, typically user “ubuntu” on earlier releases (or whichever user you added on first boot). You can then use “sudo” to mv the rc.local somewhere else, and “sudo touch /etc/rc.local” to create an empty file in its place.

Thanks for the response, I went ahead and tried to connect over serial via the micro usb. I was able to get a terminal running; however, it seems that the terminal needs CBoot to exit before it can do anything. Since the rc.local file runs at the end of Cboot the serial terminal never gets access.

Do you think theres a way to mount the file system on another computer? I vaguely remember mounting the file system on the host computer during the set up process. If I can mount the file system then I can make the necessary changes. I’m just unsure what putting my device into forced recovery mode will do.

rc.local never runs until the rest of multi-user.target finishes. If CBoot is failing, then it won’t be due to rc.local. I would be curious to see what logging is visible from serial console.

Although you cannot directly mount the filesystem on another computer, you can clone the rootfs (which produces both a mountable raw loopback image and a non-mountable sparse image), edit that clone on the PC, then reflash using the edited clone as rootfs instead of regenerating a default rootfs. Be warned that this takes a lot of time and disk space.

The short story is that if you clone and name the clone file “backup.img”, then you will also get “backup.img.raw”. Throw away the “backup.img” and keep the “.raw” version. See:
https://devtalk.nvidia.com/default/topic/1048747/jetson-agx-xavier/cloning-xavier-with-jetpack-4-2/post/5322540/#5322540
https://devtalk.nvidia.com/default/topic/1039548/jetson-agx-xavier/xavier-cloning/post/5330276/#5330276

As to the actual clone operation, basically you put the Xavier in recovery mode with the correct USB cable just like you are going to flash. If you’ve run JetPack or SDK Manager before, then you will have a “Linux_for_Tegra/” subdirectory in there, and that directory will have “flash.sh”. From that location this is the basic command for clone:

./flash.sh -r -k APP -G "backup.img" jetson-xavier mmcblk0p1

Remove “backup.img”. You can then loopback mount “backup.img.raw” (example is on “/mnt”, but you can pick somewhere else):

sudo -s
mount -o loop backup.img.raw /mnt
cd /mnt
# Explore
ls
# Go to rc.local:
cd etc
cat rc.local
# Edit this or do as you want, save.
# You can't umount if you are in that directory:
cd /where/ever/flash.sh/is
umount /mnt

Now if you want to flash this edited clone:

# This will take a lot of time...the file is around 28GB...two copies implies 56GB...check "df -H ." prior to doing this so you know how much space your host PC has:
cp backup.img.raw ./bootloader/system.img
# Reboot the Xavier back into recovery mode...you can't clone and flash without reboots between.
sudo ./flash.sh <b>-r</b> jetson-xavier mmcblk0p1

(the “-r” is very important)

Note that if the rootfs is a non-default size you may need to explicitly set the size. Size is set either in “MiB” (1024 x 1024) or “GiB” (1024 x 1024 x 1024). If your backup.img.raw is evenly divisible by 1024 three times, then this is GiB, or if only evenly divisible twice by 1024, then it is MiB (other values are not valid). So if my image is 28GiB I could explicitly state this:

sudo ./flash.sh -S 28GiB -r jetson-xavier mmcblk0p1

Hmm, so the only change I made before when it errored was adding my code to the /etc/rc.local file. I essentially changed my rc.local file to be this:

#rc.local
python my_script_that_hangs.py
exit 0

The output during the bootup was the exact same as normal, except at the end it had the logging to stout that I normally output from my script.

I sadly didn’t’ have time to try to recover using the way you proposed. We had a deadline coming up so I just reflashed the OS and reinstalled everything from scratch. If you want I can probably build out a script similar to mine that will cause the boot to hang incase anyone wants to duplicate the error.

Btw, this format is blocking if the python script blocks:

python my_script_that_hangs.py

Add a “&” to the end to let the script run in the background without blocking:

python my_script_that_hangs.py <b>&</b>

I faced similar situation that Xavier does not boot after I modified settings in fstab.
Above comands were useful to back up. Thanks.

Then I noticed one thing for writing image.
The command should include <b>-k APP</b> option to write image backed up with <b>-k APP</b> option.
So it should be

sudo ./flash.sh -r -k APP jetson-xavier mmcblk0p1

instead of

sudo ./flash.sh -r jetson-xavier mmcblk0p1

At first, when I backed up a root file system(rfs) image from Xavier with <b>-k APP</b> option and to write it in Xavier without -k APP option after modifiled.
But it didn’t boot.
Because of probably I wrote rfs to another area in mmcblk0p1.
flash.sh changes area to write when you use it with <b>-k APP</b> option, like this

APP_TAG+="-e s/size=1073741824/size=${rootfssize}/ ";

Then I noticed when I use <b>-k APP</b> option to back up rfs, I should use the command with <b>-k APP</b> when I write it in rfs area.

Thanks,

Were you trying to clone or write? This is the command for clone:

sudo ./flash.sh -r -k APP -G my_backup.img jetson-tx2 mmcblk0p1

Note that if your image is not the default size, then you probably need to specify size during the flash. For the raw image (clone gives you both a “backup.img.raw” and a “backup.img”, were “.img” is sparse and “.img.raw” is raw) you should be able to divide the size twice or three times by 1024. If the number is divisible twice, then that is the size in MiB; if you can divide evenly three times by 1024, then that is the size in GiB. As an example, if you cloned and got backup.img.raw with a size of 28GiB:

sudo ./flash.sh -S 28GiB -r jetson-xavier mmcblk0p1

I do not know if the APP_TAG you used is correct or not…perhaps this was valid.

Thank you for your comments.

I tried to clone and write.
Then I used this was for clone and it works well.

sudo ./flash.sh -r -k APP -G "backup.img" jetson-xavier mmcblk0p1

And I used this was for writing, then Xavier booted in my environment.

cp backup.img.raw ./bootloader/system.img
sudo ./flash.sh -r -k APP jetson-xavier mmcblk0p1

I used default size, so I didn’t use “-S” option.