Summary :
After corrupting the rootfs, my TX2 (eval board) won’t boot. Can’t access it via SSH either.
Is there a way to :
log in as root and modify files in / ? Every modified file was backed up so it might be possible to reverse it
recover the files, at least those in $HOME ?
flash the TX2 back to functionnality without erasing the files in $HOME ?
Black magic and GPIO-based tricks welcome, I’m up for it.
Full story :
My TX2 was originally flashed with JetPack 3.1 ; I’ve installed quite a lot of software since, but it has always worked without a hitch.
Yesterday, I was trying to install the software-end of a 6-camera module, the e-CAM30_HEXCUTX2 (link).
In order to make this module work, the recommended protocol was to flash the TX2 with LT R27 and a custom release package.
Since I wanted to keep my current data, I elected instead to opt for the “Upgrade method” and run a script (pasted below).
In the header comment, it specified that the TX2 should already be flashed with L47 R27.1 (my TX2 was flashed with JetPack 3.1, therefore L4T R28… maybe this is where I messed up).
The script ran successfully. The next step was to reboot, which I did.
The TX2 booted nicely up till just before the login screen : the mouse cursor appeared, then 1 second later the display was turned off. Complete blackout.
I tried to access the TX2 via SSH : no use, it wasn’t even detected.
That’s about it.
Thank you for reading and hopefully replying.
jwav
PS : The script I ran
# This script is meant to be used only when flashing the TX2 board is not an option.
# This script preserves the existing rootfs of the Jetson TX2.
# Only the camera related binaries and libraries are updated.
# Pre-requisites for this script is that the board should be already using L4T_R27.1 (aarch64)
# In case any other version of L4T is being used, the results are unknown and untested.
#
MACHINE=`uname -m`
RET=0
# NOTE : originally, all copy operations were done with `cp -f'.
# I changed every `cp -f' with `cp --force --backup --suffix=.original'
# so that every replaced file was backed up.
CP_OPT='--force --backup --suffix=.original'
if [ ${MACHINE} == 'aarch64' ]; then
echo "Installing DTB file ..."
sudo cp Kernel/Binaries/tegra186-quill-p3310-1000-c03-00-ar0330-sixcam.dtb /boot/ $CP_OPT
echo "Installing Kernel Image file ..."
sudo cp Kernel/Binaries/Image /boot/Image $CP_OPT
echo "Installing Kernel modules ..."
sudo tar xaf Kernel/Binaries/kernel_supplements.tar.bz2 -C / --warning=no-timestamp
echo "Installing camera module drivers ..."
cp Kernel/Binaries/ar0330.ko /home/nvidia/ $CP_OPT
echo "Installing helper scripts ..."
cp Rootfs/init_hexcam.sh /home/nvidia/ $CP_OPT
echo "Installing boot file "
sudo cp Rootfs/extlinux.conf /boot/extlinux/extlinux.conf $CP_OPT
echo "Installing conf file "
sudo cp Rootfs/modules /etc/modules $CP_OPT
echo "Installing Application ..."
sudo mkdir /usr/local/ecam_tk1 -p
sudo cp Application/ecam_tk1_guvcview/Binaries/aarch64/* /usr/local/ecam_tk1/ -r $CP_OPT
cat /home/nvidia/.bashrc | grep ecam_tk1 > /dev/null
if [ $? == 0 ]; then
echo "Path is already updated for camera application"
else
echo "#Update path for camera application binaries" >> /home/nvidia/.bashrc
echo 'PATH=$PATH:/usr/local/ecam_tk1/bin' >> /home/nvidia/.bashrc
fi
dpkg -l libsdl1.2-dev > /dev/null
if [ $? == 0 ]; then
echo "Skipping dependency installation"
else
echo "Installing Application dependencies ..."
echo "!! Make sure the board is connected to the internet before proceeding. !!"
sleep 5
sudo apt-add-repository universe -y
sudo apt-get update
sudo apt-get -y install libgtk-3-dev libsdl1.2-dev portaudio19-dev libavcodec-dev libv4l-dev libudev-dev libusb-1.0-0-dev intltool
if [ $? != 0 ]; then
echo "Make sure system time is updated properly"
echo "Make sure internet is connected."
RET=1;
fi
fi
sudo cp Application/hexcam/Binaries/* /usr/local/bin/ $CP_OPT
dpkg -l libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev > /dev/null
if [ $? != 0 ]; then
echo "Installing Hexcam Application dependencies ..."
echo "!! Make sure the board is connected to the internet before proceeding. !!"
sleep 5
sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev
if [ $? != 0 ]; then
echo "Make sure system time is updated properly"
echo "Make sure internet is connected."
RET=1;
fi
fi
if [ $RET == 0 ]; then
echo "Done! Now reboot the board to get the camera working."
echo "Refer to README.txt and e-CAM30_HEXCUTX2_Linux_App_User_Manual.pdf in case of any doubt."
fi
else
echo "Please run this script on the Jetson TX2 board !!!!"
fi
If not, then you can clone the rootfs. This creates a very large file (the size of the complete file system, e.g., 30GB), so your host must have lots of disk space and lots of time. Details on clone differ depending on release. Assuming R28.1 or R28.2, then you’d go to the driver package directory (which JetPack may have put in place if you used that)…the subdirectory name is “Linux_for_Tegra/”. Find file “flash.sh”. Put your Jetson in recovery mode, make sure the micro-B USB cable is connected…verify host can see the TX2:
When done you will have files “backup.img” and “backup.img.raw”. The “.raw” file is the one you want…the other one is “sparse” and can be used to flash with, but cannot be used in any other way.
You can mount the raw file on loopback and explore it…you could edit this…and then flash the rootfs to the Jetson with the modified version as a fix. Be very careful…you will want to use “-r” to avoid overwriting the image…this is the “re-use rootfs file system.img” option. If you place either the sparse or raw file as “bootloader/system.img”, then flash will work with this. Example after replacing system.img with your file:
sudo ./flash.sh -r jetson-tx1 mmcblk0p1
An example of loopback mounting on “/mnt”:
sudo mount -o loop /where/ever/it/is/backup.img.raw /mnt
cd /mnt
ls
# explore, edit, so on
cd
sudo umount /mnt
sudo cp /where/ever/it/is/backup.img.raw /some/where/else/Linux_for_Tegra/bootloader/system.img
# ...then do the flash of rootfs as mentioned above.
The serial console and cloning methods were the 2 ways I previously identified as possible solutions, but I didn’t know much about it.
Reading your post convinced me to go ahead with the cloning method today (no serial cable yet). I’ll write a report afterwards and hopefully mark your reply as the answer.
I’ve tried and failed the cloning method with an Ubuntu-14.04.05 x64 image run with VirtualBox 5.1.34 r121010.
The Jetson is placed in recovery mode and detected with lsusb.
The following command is run (the result is pasted below):
I’ll try the serial console trick when the cable arrives. It might have failed because I did it from a virtual machine, but I don’t have an Ubuntu x64 station with internet access lying around anyway.
./tegraflash.py --chip 0x18 --applet "/home/jwav/jetpack/64_TX2/Linux_for_Tegra_tx2/bootloader/mb1_recovery_prod.bin" --cmd "dump eeprom boardinfo cvm.bin" --skipuid
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
[ 0.0012 ] Generating RCM messages
[ 0.0022 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 --download rcm /home/jwav/jetpack/64_TX2/Linux_for_Tegra_tx2/bootloader/mb1_recovery_prod.bin 0 0
[ 0.0031 ] RCM 0 is saved as rcm_0.rcm
[ 0.0031 ] RCM 1 is saved as rcm_1.rcm
[ 0.0031 ] List of rcm files are saved in rcm_list.xml
[ 0.0031 ]
[ 0.0031 ] Signing RCM messages
[ 0.0039 ] tegrasign_v2 --key None --list rcm_list.xml --pubkeyhash pub_key.key
[ 0.0081 ] Assuming zero filled SBK key
[ 0.0082 ]
[ 0.0082 ] Copying signature to RCM mesages
[ 0.0094 ] tegrarcm_v2 --chip 0x18 --updatesig rcm_list_signed.xml
[ 0.0105 ]
[ 0.0106 ] Boot Rom communication
[ 0.0114 ] tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml --skipuid
[ 0.0123 ]
Error: Return value 8
Command tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml --skipuid
Reading board information failed.
First, I followed JetsonHack’s tutorial exactly, which led me to a terminal on the Jetson allowing me to use bash as if it were an SSH terminal.
I inserted a USB key into the Jetson, mounted it, and abducted the files I needed to save from the flood.
=> Mission accomplished. I’ll tag linuxdev’s poste as the answer since it contained the solution as well as an alternative. However, I’d recommend to anyone having the same problem to try the serial console method before the cloning method, since the former is considerably less time-intensive.
I’ll now go ahead and try to mend my damaged rootfs to see if it works.
Using a VM will cause flash and clone to fail. Some people have got this to work, but usually it doesn’t just work out of the box. When it comes time to flash again there is a high probability that the flash will fail due to USB issues. VMs are not universal substitutes when it comes to how USB enumerates.