Jetson TX2 freezes after flashing the JetPack 3.2.1

Hi all,
I just started to flash my new Jetson TX2 with JetPack 3.2.1, the installation on my host laptop reaches to a point to flash everything through the USB cable and wait to find the Jetson IP address. First I was trying to use WiFi but then I connected the Jetson directly to the router, although it shows it’s connected but my host laptop stills waiting to acquire the IP.

The problem is the Jetson boots up and stays in tegra-ubuntu page but neither mouse nor keyboard works to let me enter the password. It seems to be freezing in this page.

FYI, flash stage is independent of software additions. The flash occurs with the Jetson in recovery mode, and runs over the micro-B USB cable. Once flash completes the Jetson reboots and USB is no longer used…extra packages depend on wired ethernet.

If the flash worked, then in JetPack you can uncheck flash and run extra package steps at any time using only the wired ethernet if the Jetson is up and running. One catch is that flash creates a one time mechanism to report the IP address to JetPack on the host. Without this you must watch the console and any popup windows to see when it asks for the IP address (and it can be easy to miss JetPack asking for this). Flash cannot move forward until the IP address is known.

That is for normal use, but if the keyboard and mouse don’t work, then something else might be wrong. First of all, was your host PC running a real Linux install and not a VM? Second, was the disk partition you ran from using a native Linux file system, e.g., ext4, and not something like a live DVD or thumb drive with VFAT or NTFS? Next, the amount of disk space required is enormous because you are building an entire file system on the host, so did you have enough space? One way to figure that:

df -H -T /where/ever/you/ran/flash/from

Video failing might also be a different problem from what you think it is. For example, the monitor must be HDMI or using an adapter to a monitor capable of producing the EDID data (most DVI would qualify, 100% of 15-pin D-Sub VGA will fail, DisplayPort will work). If you have a router assigning an address to the Jetson, and if you can find the address assigned, then try to ping that address (even better yet, try to ssh to that address and log in). If not, then perhaps CTRL-ALT-F2 will get you to a text-only console. Beyond that you need a serial console cable to see what is going on. For serial console see:
http://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/

Thanks for your prompt reply.

Before anything else, I have to say I’m a newbie to Linux and Ubuntu so I may ask simple/obvious questions throughout the process.

Based on your description, I guess the flash has not been completed as the IP address was not known. I connected the Jetson to a router and started the JetPack in custom mode to not reinstall the packages on my laptop, but when the Jetson goes to recovery mode and started flashing I got the following error:

“Failed to flash device. Check:~/flash_os_tx2.log for more details.”

When I open the log, I get this message at the last lines:

“Making system.img…
rm: cannot remove ‘mnt’: Device or resource busy
clearing ext4 mount point failed.”

@linuxdev I followed your advice from other posts and solved the “mnt” folder issue. Now when I run the JetPack it shows the flash has successfully been done and starts looking for IP address but cannot find it.
My host laptop is connected to WiFi and the Jetson board is connected to the Wi-fi router with Ethernat cable. Is there any way to find Jetson IP and manually enter it to finish the flash?

There are things which can go wrong after a successful flash such that JetPack still does not find the correct IP address. Examples are firewall, proxy, or router issues. It is somewhat common to see questions on this forum due to lack of IP address in the context of both failed flash and successful flash. According to the last of the logs the flash was successful and the most likely failure is network setup. The first failure log was because two attempts were made to use “/mnt” without removing the first user of “/mnt”…two processes can’t control “/mnt” at the same time. JetPack itself won’t have this issue, but something you did before used “/mnt” and then didn’t release it. If you mounted something there and you did a “cd” to go there and look around, then you need to “cd” out of the directory before it can “sudo umount /mnt” (the command to unmount is spelled “umount”).

Beware that JetPack only sets up for wired ethernet…the setup of WiFi after flash is beyond JetPack since it involves the router and SSIDs. This would cause additional package install to fail even when flash succeeds. For this reason only wired ethernet is supported for package installs. However, read on…

Despite WiFi not being supported it is possible for it to work. You would have to log in to the Jetson after the flash and manually set up WiFi, and then guarantee your host PC can access the IP address using ssh to the “nvidia” account. If your IP address turns out to be something like “192.168.1.3” (a contrived example), then your host could verify the address with this on a text console:

ping 192.168.1.3

Or, you could log in to the Jetson’s “nvidia” account via (default pass for use “ubuntu” is “ubuntu”, and default pass for user “nvidia” is “nvidia”…something you will want to change after you are done):

ssh nvidia@192.168.1.3

Note that first contact to another system from the host will prompt you to answer yes or no as to whether to trust the connection, and you would need to say yes. After that this IP address should work.

As for finding the IP address, if you log in directly to the Jetson (either the “nvidia” or “ubuntu” account), then run these commands and show the result after WiFi has been configured:

# For wired ethernet:
ifconfig
# For WiFi:
iwconfig

The actual WiFi configuration is the same on this Ubuntu as any other Ubuntu…I mention this because any topics you see on the internet for Ubuntu 16.04 WiFi will be valid. I personally attempt to avoid WiFi and will not be the best person to give advice on its setup, but the tool bar along the top will have two icons you are interested in. The first is the icon with the two arrows (one up, one down…for enabling and disabling networking), and the other with the gear (for a control panel if you click the “system settings”, then “network”).

If you plan to be doing any development on the TX2, then you will likely want to invest in a serial console cable early on. These are inexpensive. See:
http://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/
(serial console functions well even when a large part of the system is missing or failed)

linuxdev,
Thanks for your detailed reply.

I’m not using Wi-Fi anymore based on your advice. I connected both my laptop and the Jetson to a router and did the flash multiple times. After the flash is complete, I get the following message every time:
Host 192.168.0.XXX not found in /home/kash/.ssh/known_hosts

Then it starts debugging and at the end it writes:
nvidia@tegra-ubuntu: ~$ b /home/nvidia/.ssh/authorized_keys
nvidia@tegra-ubuntu: ~$ exit
logout
Connection to 192.168.0.xxx closed.

Then the Jetson reboots and it freezes in the tegra_ubuntu page.

I am assuming you are running JetPack under the user “kash”? The “known_hosts” file it is mentioning only gets created on the first connection. This file then gets appended to each time a new connection is accepted as valid. Perhaps JetPack is just confused by an account which never used ssh before.

It is harmless to do the following to create “known_hosts” if the file does not already exist (and this will not harm the file if it does exist) while logged in as that user on the PC:

touch ~/.ssh/known_hosts

So here is a way to know for certain about ssh issues…boot up the Jetson and then try to “ssh nvidia@whatever_IP_address_it_is” (and of course put in the IP address as something like 192.168.1.3…whatever the address really is for that Jetson…login success by that implies you are ready for JetPack). Once your host under account “kash” is able to manually connect via ssh you should be able to run JetPack again (unchecking flash and just checking package steps…using ethernet without recovery mode and not USB since you are not flashing again), and then those steps should no longer interfere.

If for some reason manual ssh does not work, then debug steps for command line ssh would also be the same debug steps which you would need to do for JetPack. An example is “ssh -vvv nvidia@whatever_IP_it_is” (a number of ssh steps would normally fail and you would get a lot of output, but at some point it should succeed…we are looking for why a password login fails if it does fail all the way to the end).

linuxdev,
I was able to use “ssh nvidia@whatever_IP_address_it_is” and connect to the device manually.

Then I use cd command in a new terminal window and run the JetPack without flashing (of course without usb). It started installing CUDA 9.0 on the Jetson and said it is successful. Then it stopped after that. I guess it is supposed to install the rest of the package but it does not continue.

What you can do is run the software addition steps just as you already have since it won’t hurt to tell it to install again. Perhaps there was some dependency it wasn’t automatically checking and never tried to install more than CUDA. There is a complicated set of dependencies when adding the related packages, and the CUDA repo followed by CUDA itself are the starting point to all of the others, so you can consider it progress.

Something you can do just to see installed packages (in this example anything with “cuda” in the description) would be:

dpkg -l | grep -i cuda

(you can adjust that for other package names as well, e.g., if a package has “tensor” in the name, then use that)

See if another attempt will add more packages.

The packages I’m going to install are:
CUDA Toolkit
Compile CUDA Samples
cuDNN Package
TensorRT
OpenCV
Multimedia API package

None of those packages can be installed without selecting and installing the CUDA Toolkit.So, if I understand your recommendation correctly, I have to install the CUDA Toolkit by connecting to the Jetson using SSH command and its IP address, then after the installation is finished, I need to do the same procedure to install the second package and so and so for. But the problem is when I start installing the second package, I cannot select “no action” for CUDA Toolkit (because of dependencies), and if I select “install” for CUDA Toolkit, then it only install the CUDA and then stops.

btw, when I connect to the Jetson using SSH command and type “dpkg -l | grep -i cuda” command, nothing happens. I tried with TensorRT too but had no success.

Apart from installing the packages, I assume Jetson should allow me to log in the tegra_ubuntu page. why it freezes in the log in page? Is it related to the packages?!

Thanks for your help,
–Kash

Update: Just flashed the Jetson with JetPack 4.2 and Ubuntu 18.04 successfully!

Tried to re-flash it with JetPack 3.2.1 and it got stuck in:
“Finished Flashing OS
Determining the IP address of target…”

I know what is the IP address, so I tried it using the SSH command I was able to make connection with Jetson, but for a reason the installation is not able to acquire the IP.

–Kash

Typically when this happens there is some obscure place to enter the address and once that is found and the IP entered it works. Glad to hear it works though…JetPack4.2 is going to be a far better choice in the long run anyway.

I did not find any place to enter the IP for 3.2.1.

So is there any solution for me to install the 3.2.1 rather than 4.2? Our software team has developed their software in 3.2.1 and I need to have 3.2.1 and Ubuntu 16.04 installed.

It has been a long time since I looked at 3.2.1, but typically there is a console at the bottom which you need to click on a tab to select before you will see it. Look at every tab related to the console…this is the most likely way of proceeding.

If that doesn’t do it…

In the past I think sometimes ssh itself asks for a pass and this might be mistaken for needing an address…in which case you need an “askpass” program on the host itself. For a list of askpass possibilities on the host:

apt search askpass

The most basic:

sudo apt-get install ssh-askpass

If the location of your askpass is “/usr/bin/ssh-askpass”, then you can be sure the program knows about it via “export SSH_ASKPASS=/usr/bin/ssh-askpass” in the same terminal you start JetPack from. Possibly this would have already been set, but this method guarantees askpass won’t be the problem. If the issue really is one of IP address, then it means askpass could not have been the issue (but I have a distrust of error messages…they don’t always tell you the correct error).

Thanks for your help, but I’m still having problem with 3.2.1.

I decided to go with 4.2 which seems to be working ok.

Thanks again for your support,
–Kash