How to reach Jetson after flashing memory

Robotics & Edge Computing Jetson & Embedded Systems Jetson TK1
1

Brand-new Jetson TX1; followed setup instructions to flash Jetson memory from Linux using Tegra210_Linux_R24.2.0_aarch64.tbz2 (from Nvidia site)

When Jetson is connected to monitor and keyboard, every couple of minutes it prints out a lot of messages on black background, looks like it is saying it is OK

But operating system never comes up - no Ubuntu screen

Both green lights are on, on Jetson

Connected Jetson to Ethernet port on router
Router shows that Ethernet port is up, but cannot ssh, ping, etc. and Jetson does not show up on router’s page of DHCP clients

I suspect that by flashing memory, I overwrote the Linux operating system that was there (naively)

How do I get access to the Jetson again?

2

Flashing should install Linux so your steps should have been harmless. There is more than one way to flash though, and more than one option. R24.2 is what you used, and this is the current recommended release. A failed flash could be repeated without any harm to the Jetson.

Did you use the JetPack GUI installer front end, or did you use the flash.sh script on command line (JetPack actually uses flash.sh as well)? If you used flash.sh what was your exact command line, and did you flash with sudo (I don’t think you need to start JetPack sudo, but flash.sh must be run sudo)?

Can you describe if your router perhaps has some form of firewalling or security which might deny an unlisted MAC address?

Can you give an example of any of the “ok” listing you see? I ask because the text console output of a properly booting system normally does this…flash may have been successful even if some part of boot is failing.

3

Thank you SO much, @linuxdev. I"ll do my best to answer these questions.

The command I used was
sudo ./flash.sh jetson-tx1 mmcblk0p1

and during the night, I also tried this:
sudo ./flash.sh EBT jetson-tx1 mmcblk0p1
(I am not sure it was EBT but it was 3 letters in caps)

The router does not have any firewalling against MAC addresses that I know of.

I moved it back to connected to the monitor:

647221] CPACR_EL1 = …
CPU3 Status: online
ARM_x29 = 0xfff…
vi vi …
Tegra…

it does this every minute or so

The green lights are both on, one near power, the other next to it

It scrolls for a while and I don’t have scrolling back

I expected it to come back to the Ubuntu “front screen” but it never has

Right now it is connected to Linux, but not to Ethernet

4

There are lots of other lines, it seems to be a sort of dump of its processes
There are other lines about mmc:

I was expecting it to just come back to me and put up the Ubuntu screen, so that I could configure WiFi, go on to JetPack and development, etc.

Mouse is still connected, keyboard still connected via USB

I am also curious about a small thing that looks like a memory card, which was in a bag, in the kit.
I didn’t know if I was supposed to do something with it.

5

Although probably unrelated, you might be interested to know the default flash command won’t fully use the eMMC. I’d recommend:

sudo ./flash.sh -S 14580MiB jetson-tk1 mmcblk0p1

You should probably avoid the EBT flash. I would flash again with the above command line.

One thing which will cause an issue even if hardware is all OK and even if flash is done correctly is if the file system image being flashed is truncated. The host needs a lot of hard disk space since the entire file system is being generated and then a compressed copy created in addition to that. On your host, in the directory where you ran “flash.sh” script, what is the output of this:

df -H -T .

Also from that directory notice a “bootloader” subdirectory. A flash generates two files there which are of interest. The smaller file is approximately 2GB in size, and all that is needed is to verify it exists. That is “bootloader/system.img”. A second (larger) file which is uncompressed can be validated by knowing its exact byte size…what is the exact byte size of “bootloader/system.img.raw”?

6

@linuxdev, I am very grateful. I so want to make this beast work!
Thanks again for your quick responses.

This is a TX1, so it has to be jetson-tx1, and the rootfssize is being rejected at the command line.
It says, Invalid rootfssize

joel@warrior:~/nvidia/Linux_for_Tegra$ df -H -T .
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 157G 20G 129G 14% /

joel@warrior:~/nvidia/Linux_for_Tegra/bootloader$ ls -lt system.img.raw
-rw-r–r-- 1 root root 15032385536 Nov 10 18:43 system.img.raw

7

I computed from the number in the system.img.raw that it is 14336 MiB, and I started the ./flash.sh command, after putting it into Force USB Recovery mode.

I think you are confirming that I’m on the right track, yes?
In other words, I should reasonably expect that it comes back to the Ubuntu screen?
It sounds like I did not do anything “wrong” - but that the flashing just hasn’t yet worked all the way.

8

it is saying “Reset the board to boot from internal eMMC”

in the meantime, the screen output continues.
what is the best way to reset the board to boot from internal eMMC?

Thanks…

9

I skimmed through too fast and assumed TK1 because this is the TK1 forum. Other information previously mentioned though should be interchangeable. Just use “jetson-tx1” where I had “jetson-tk1”. If interested the thread could be moved to the JTX1 forum:
https://devtalk.nvidia.com/default/board/164/jetson-tx1/

Your host file system should be fine in terms of both type and free space.

The system.img.raw size found matches flash.sh parameter does match “-S 14336MiB”. This is unusual, but it is close to full use of eMMC and should work. However, rejecting “-S 14580MiB” is of concern, this should not be the case. “-S 14580MiB” works unless there is an error. Please try this again:

sudo ./flash.sh <b>-S 14580MiB</b> jetson-tx1 mmcblk0p1

…whatever error it gives would be of interest.

I can confirm that basically what you are doing is correct. Sometimes a host that is a VM gets odd errors, but it doesn’t sound like your host is a VM.

When flashing is complete from a command line flash.sh it should restart on its own. The command to reset and boot is a normal and expected response from a successful flash. If for some reason it fails to reboot on its own you could just press the reset button (the one furthest from the power button in that row of buttons, label “RST”).

When it starts booting see if you can find an IP address on your router, or at least see if the query went out (I’m hoping an attempt to flash with “-S 14580MiB” works…if so, then DHCP request should go out).

10

Well, this is odd, now, when I run the ./flash.sh command with the 14580 MiB, it accepted it.

I put it into recovery mode, in order to do the flashing.

Lots of this:

3.3730 ] Flashing the device
[ 3.4198 ] tegradevflash --pt flash.bin --storageinfo storage_info.bin --create
[ 3.4242 ] Cboot version 00.01.0000
[ 4.0659 ] Writing partition GPT with gpt.bin
[ 4.0691 ] […] 100%
[ 4.0754 ] Writing partition NVC with nvtboot.bin.encrypt
[ 4.6451 ] […] 100%
[ 4.6630 ] Writing partition APP with system.img
[ 4.6958 ] […] 100%

and then it appears it is OK:

[ 52.3072 ] tegradevflash --reboot coldboot
[ 52.3132 ] Cboot version 00.01.0000
[ 52.4160 ]
*** The target t210ref has been flashed successfully. ***
Reset the board to boot from internal eMMC.

but the screen just continues to flow, every minute or so, with the data that looks like dumping data.

I’ll plug it into the router, in the state it is in, and see if there is any response.

Why don’t I see the Linux Ubuntu screen?
(I guess that’s maybe what we are trying to work out)

Thanks…

11

When it is connected to Ethernet, and it is writing to the monitor, I can see the light on the router go ON.
The Ethernet light on the Jetson is also lit, at that time.

After the writing to the monitor finishes - the light goes back off.

The router does not show this device in the DHCP clients table.
I liked your point about “unlisted MAC address” but I don’t know what that means.

12

The flash itself seems to have completed normally. The data might be from errors, or just bootup scrolling. Earlier you showed this:

CPACR_EL1 =

…which I suspect is not normal, but was hoping a fresh flash made things work. EL1 is a reference to the kernel mode interrupt and might be mentioned in an OOPS or other crash. The part with “arm_x29” I believe refers to one of the registers in ARMv8-a (which is the mode in EL1 interrupt…I would also not expect this to be visible except under error conditions).

I’ll mention a very small possibility since it looks like the board is performing as expected during the flash (a full flash log would be needed to verify this, but odds are high the flash was completely normal based on the most recent information). The sample roofs goes into the rootfs subdirectory and must be unpacked by root (or sudo)…this plus the flash script making edits to boot time files are how the image is created on system.img.raw (which in turn populates sparse/compressed image system.img). I’ve seen accidental population of the rootfs subdirectory by things not expected. This would propagate to the system.img.raw and be flashed to the Jetson root partition without any knowledge by the flash program that it isn’t a valid rootfs. The one example I just mentioned would be if sample rootfs was not unpacked by root, which would make invalid permissions; this won’t be the case based on current issues. What has happened and could result in a bizarre boot even though flash and hardware are valid is that sometimes people use a utility like wget to download files…and they end up downloading the html page content instead of the rootfs…the rootfs ends up with nonsense and boot might spew out some odd information.

Describe more closely what you can about what you do see when it starts to boot. You could even snap a picture of it with a camera and post it. I’d like to see if it is error information from a Linux kernel, ordinary boot scroll, or nonsense from something else.

FYI, it is quite common for people to have a valid and working boot but not have video function correctly. This gets mistaken for a crashed system or a bad flash when it is really a video configuration problem. Seeing a DHCP request validates the idea that the flash was not only successful, but that the system is also running even if it can’t be seen. Being able to ssh in would be of great help if there is a video issue to resolve.

You may want to look around in the rootfs subdirectory and see if it looks like a real Linux root file system tree. You could also mount and look at the actual image by loopback mounting the system.img.raw and you’d see the absolutely exact copy of what the Jetson had flashed to the root partiion. E.G.:

sudo -s
mount -o loop /wherever/the/image/is/system.img.raw /mnt
cd /mnt
# ...do stuff like "ls"...
cd
umount /mnt
exit

The purpose of this is to confirm that what went into the Jetson is correct, which would in turn suggest whether there was an installed software issue versus a hardware issue.

And lastly, a serial console is invaluable in these situations. It’s just a string of bytes over a UART and there are no real drivers to fail. Video can fail, console can fail, networking can fail…and serial console still works. Serial console is so simple you can even interact with u-boot prior to the Linux kernel taking over. If you can try this here is a URL on wiring for serial console:
http://elinux.org/Jetson/TX1_Serial_Console

13

Hi, I tried the mount, and it let me look around, boot, etc, home, lib, usr, var.

I’ll try to snap a picture right as it begins. It goes by quickly and I don’t know how to scroll or collect it.

I have never done wiring like that - for serial console. A little intimidated.

I’ll try the picture coming up…

3.jpg
3.jpg3024×4032 3.23 MB

2.jpg
2.jpg3024×4032 3.93 MB

1.jpg
1.jpg3024×4032 4.41 MB

4.jpg
5.jpg
5.jpg3024×4032 4.33 MB

3.jpg
3.jpg3024×4032 3.24 MB

2.jpg
2.jpg3024×4032 3.94 MB

1.jpg
1.jpg1512×2016 1.42 MB

4.jpg
4.jpg3024×4032 3.54 MB

14

The pictures are numbered 1-5.
I have a 6th one which is what it looks like at the end.

It does look like it is trying to reboot…

I have a better camera here with a faster flash speed if that’s needed.

15

Up until the end of 4.jpg boot is normal. Boot loader worked, kernel loaded, init began, most everything was set up. The system then did a recovery on the file system during mount, which isn’t really a problem, but it shows it may have been shut down uncleanly rather than with a proper shutdown command (understandable with these circumstances). Normally file system issues like this are handled via restore of the journal.

5.jpg is another story. This is a full kernel OOPS. Perhaps there was something the ext4 file system journal didn’t recover, but odds are something else is the issue…I wouldn’t say journal recovery not being enough is truly “unlikely”, it should be kept in mind, but that kind of long OOPS message is more or less unrecoverable. The thing about this is that boot had more or less completed in text mode without issue. Video failing with a serious failure could be the cause. In addition to this, as Linux tries to spawn graphical mode and fails, it’ll then retry over and over. This could be the flashing you see…going to console mode which works right, then attempting to go to graphical mode which crashes, and then back again, over and over.

If you could get into the system via ssh or serial console there are some things to look at, but one step of flashing (which is actually preparation and not actual flash) could cause this by not having all of the correct files installed with correct permissions.

During preparation for use of flash.sh it is required to unpack the sample rootfs with sudo. This probably isn’t the issue. Right after this you cd back up one directory and use sudo to run “apply_binaries.sh”. If this latter step was not done it could cause graphical mode failure. Was this latter step done with sudo?

Basically you have a working system so long as you don’t go to graphical mode.

16

Thanks again. It does sound like you explained what’s happening - crashing over and over trying to go to graphical mode.

I will try to re-do the apply-binaries.sh and all of it, again. I’ll write back after I try that, but I thought I did it right the first time.

Let me be sure I understand - I have a working system which can’t go to graphical mode; but I also can’t ssh or join my home network. The machine isn’t usable - it seems like.

Is there a way to tell it not to go to graphics mode?
I really don’t mind if I interface on command line, for now. But I don’t know how to tell it to do that.

And here is another question - should I not have flashed memory?
should I have set up network accessing before trying a flash?

17

Ubuntu to me does weird things for networking, and some of it isn’t set up until the GUI runs (this is nonsense, but I guess it is to personalize wireless).

You can flash using an edited system.img.raw and tell it to default to text mode. Here is how…

“In the old days”, there was no compressed version of the image being flashed. It took a long time to flash. There was only system.img, and system.img.raw did not exist. The flash program then and now is capable of flashing correctly whether the image is compressed (sparse) or not (raw). What happened was that system.img was created uncompressed, and still is…but now system.img gets moved to the name system.img.raw, and system.img is created from this. Be careful to not overwrite your raw image if you want to edit…but it is about 15GB in size…so backups will be slow to copy.

Rename system.img.raw so flash.sh can be kept from overwriting it without copy of the entire file (edits to be explained). For example, in the bootloader subdirectory:

cd /wherever/it/is/Linux_for_Tegra/bootloader
sudo -s
mv system.img.raw saved.img
cp saved.img system.img
mount -o loop ./system.img /mnt
cd /mnt/boot/extlinux
# ...now edit extlinux.conf with your favorite editor...
# ...save the edits...
cd -
umount /mnt
cd ..
./flash.sh <b>-r</b> -S 14580MiB jetson-tx1 mmcblk0p1
exit

The “-r” parameter tells flash to not generate a new system.img, and instead to reuse the one there now. It just happens that this is the raw image renamed to system.img and edited while it was loopback mounted. Here’s the edit info…

Within extlinux.conf there is a very long line which is the “APPEND” key/value pair. Line wrap will make it look like it isn’t one line, but it is…and it must stay one long line with space separated values. At the end of that APPEND line add a space then:

systemd.unit=multi-user.target

So long as that particular extlinux.conf makes it in boot should stop at console mode. It may be possible at that point to work directly on the Jetson without crashes getting in the way. Once things are fixed you could remove that edit directly on the Jetson and not worry about flash.

Incidentally, when in console mode, the reverse command to switch to graphical mode is:

sudo systemctl isolate graphical.target
18

Thanks again. However, there were 2 problems.

At first, it would not accept the 14580 size on the -S, saying something about 512 block size.
I removed the -S.

Then, when I tried your edit, and unmounted, and tried the flash.sh again, i get this:

Warning: missing eksfile (bootloader/eks.img), continue…
copying bctfile(/home/joel/nvidia/Linux_for_Tegra/bootloader/t210ref/BCT/P2180_A00_LP4_DSC_204Mhz.cfg)… done.
File “/home/joel/nvidia/Linux_for_Tegra/elf-get-entry.py”, line 73
print ‘0x%x’ % ep
^
SyntaxError: Missing parentheses in call to ‘print’
Could not determine entry point of bootloader binary

Thanks…

19

I should have added, the original error, on trying with the 14580 MiB, was his:

Making system.img…
Error: ext4 file system size has to be 512 bytes allign.

20

Hi, I tried from a different Linux (CentOS) and I got the same problem (boot interrupted by the OOPS)
Then I modified the image, as you suggested, with the edit,
and it pushed it over to the Jetson, but unresponsive after that.

Any thoughts? At this point, do I go ahead with an RMA?

Thanks…