Hello,
we have 4 Jetson Orin nano developer kits and 4 SanDisk SD Cards on which we flashed ubuntu 20.4.
Everything worked fine first, but after some time, the Jetson will just refuse to boot. It shows the nvidia splash screen, says it attempts direct boot and the display stays black for several minutes. Then it tries to reboot and this cycle repeats forever.
When we flash a new SD card or even reflash the old one, everything works totally fine again. Until one day, it doesn’t. This problem can occur on any of the 4 Jetsons with any of the 4 SD cards.
It seems like the Jetson will corrupt the SD card eventually. However, if we put the SD card into a laptop running ubuntu, we can view the content of the card and even copy our files. Nothing indicates that the card is totally corrupted.
I hope someone can help us, thanks in advance.
You (or someone from NVIDIA) might want to move this to the Orin Nano forum (this is the old Nano forum, based on the TX1). Orin Nano is here:
https://forums.developer.nvidia.com/c/agx-autonomous-machines/jetson-embedded-systems/jetson-orin-nano/632
Which L4T release are you using? See “head -n 1 /etc/nv_tegra_release” (FYI, L4T is what gets flashed, and is Ubuntu plus NVIDIA drivers; JetPack/SDK Manager are the flash tools on the host, and versions are tied together). Also, can you verify this is a developer’s kit? If this is a module on a third party carrier board it changes a lot in boot stages. Assuming this is a developer’s kit (no eMMC for an Orin Nano), then it means there is QSPI memory on the module itself which has the boot content, plus the software equivalent of a BIOS. The earlier releases of that software have a lot of fixes when moving to the most recent version. You can flash the module itself, and generally speaking, I wouldn’t consider creating an SD card to be flashing. If you flashed, was it the module’s QSPI, or was it the SD card?
Basically, what I’m getting at is the need (if not the most recent release on the module) to flash the most recent release to the module, and make sure the content on the SD card is designed for that QSPI release (assuming it is not a third party carrier board).
Hi lugh.martensen,
Could you share the result of the following commands on your board?
$ cat /etc/nv_boot_control.conf
$ cat /etc/nv_tegra_release
What do you mean about “some time”? How long is it? Or do you run any application or do any modification in rootfs?
Could you share the full serial console log when it gets broken(can’t boot up)?
How did you flash the board and SD cards?
I will look at the release version on monday, I don’t have the boards with me right now.
Yes it is a developer kit, to be precise, it is this one: JETSON ORIN KIT: NVIDIA Jetson Orin Developer Kit, 6x 1,5 GHz, 8 GB DDR5 bei reichelt elektronik
I flashed an SD card and inserted it into the Jetson using this tutorial: https://developer.nvidia.com/embedded/learn/get-started-jetson-orin-nano-devkit#write
Not sure if the SD card image is designed for the QSPI release, how do I check that? I just downloaded the Image for Jetson Orin nano as described in teh tutorial I linked
I actually don’t know how to check if the QSPI is for a given release (that’s a good question, I’d be interested in knowing that myself…how to check QSPI release on a running Jetson). Usually though anyone purchasing any SD-card model dev kit is advised to flash that QSPI once at the start. I will suggest that you can (or should) start by flashing the module itself to a current release since we don’t know what release it has, and because this is most likely part of why boot would hang.
Here is the result of the two commands.
We run ROS2 foxy on it with a few programs. We use the Jetson for a f1tenth car, basically an RC car with a Jetson attached. Here is their build website: Build (We use Jetson Orin nano dev kit though). I am not sure if ROS does any modifications in rootfs.
Its hard to say when it happens. We use the Jetson for approx. 5 hours on three days a week. After about two weeks, problems started to accour. However, one Jetson already failed after one day.
I did not flash the board, only the SD card. I used this tutorial: https://developer.nvidia.com/embedded/learn/get-started-jetson-orin-nano-devkit#prepare
As described in the tutorial, I used balena etcher to flash the image from the website.
I should mention however, that we also used the command tool “dd” to clone the sd card into an image file so I could flash that file to other cards, as described in this tutorial: https://jetsonhacks.com/2020/08/08/clone-sd-card-jetson-nano-and-xavier-nx/
I do need extra hardware to read out the full serial concole log, right?
Do you maybe know a good tutorial on how to do that? I googled it but could not find anything official. It’s the first time I setup a Jetson from scratch, I thought I could just flash the image onto the SD card and I’m good to go.
I found out the release version btw, you can see it in the foto that I attached to the reply for @KevinFFF 's message
Is it using the Orin Nano devkit on the product?
Could you try to use SDKM to flash the board and check if it could recover?
Yes, you need a serial console cable. Please refer to the following instruction to capture the serial console log and provide it for further check.
Jetson Nano & NX Style - Serial Debug Console - JetsonHacks
Documentation is from the same web page as that which is for a given L4T/JetPack release. Note that L4T is what actually gets flashed, and is Ubuntu plus NVIDIA drivers; JetPack/SDK Manager is the flash software which goes on the host PC. Their versions are tied together, so if you pick one, you’ve picked the other. If you go to the web page for either you end up on the same web page, which has the correct documentation for that release. See:
Just flash, and create an SD card, from the most recent L4T release if possible.
No, the tutorial on how to build the car did not use the Orin Nano devkit but the xavier nx devkit. I was not the one responsible for choosing the hardware, but now that we have it, it should work with it too. Especially because it works for a time and then randomly fails, so there should be a way to fix that, right?
I read out the log of the jetson when it tries to boot with the broken card. The log is saved in this txt file:
Log_with_recovery_boot.txt (132.2 KB)
As you can see, the Jetson tried to boot in recovery mode, which it usually doesn’t do when it has a working sd card installed. I tried to switch the “OS chain A status” to “normal” in the L4T configuration in the “bios” of the Jetson (The option was set to “unbootable” before). Then it attempted direct boot, the log of this attempted boot is in this txt file:
Log_with_direct_boot.txt (132.0 KB)
Can you see something usefull in the logs?
Sorry to jump in. Just want to point out one point here.
Both of the logs you just shared got truncated in each line. Please resize your terminal and dump log again.
Before you share out any log, read it first to make sure it is really completed.
If you don’t know what I mean, here is the screenshot of the log you shared.
For example, you can see that your log got cut after a specific length…
That line with “Linux version 5.10.120-tegra …” ends with a “3” but we cannot see anything else after that.
“Reserved memory” gets end with a “B” but nothing else is shown afterwards.
Hi, you are right. Here is the new direct boot file:
defect_direct_boot.txt (117.2 KB)
Also, when you wait a long time, the Jetson tries to boot again (and fails). This should be visible in this log:
defect_direct_boot_extended.txt (203.5 KB)
We have also reset the option in the “bios”, so this is the log when it tries a recovery boot:
defect_recovery_boot.txt (122.3 KB)
Hi, as @WayneWWW pointed out, the files are not complete. I uploaded an updated version in the response to his comment, you can find it there
[ 12.310568] Switching from initrd to actual rootfs
[ 12.422781] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[ 12.433530] CPU: 2 PID: 1 Comm: init Not tainted 5.10.120-tegra #1
There’s kernel panic in your case when it mount mmcblk0p1 as rootfs.
It is triggered from watchdog (120s) and it seems you hit the kernel panic.
After few trials(3 times as default), it would enter into recovery state.
Could you try using other SD card and format it as ext4 before use?
Put it in SD slot and use SDKM to flash the latest JP5.1.2(R35.4.1) to verify.
We could try formating it to ext4, but I don’t get why that would change anything. I don’t know how SDKM is doing it but when we use balena etcher, it deletes the partition anyways. So it shouldnt matter what you are formatting the card too, right?
We will try SDKM once we have set up a new computer with ubuntu 20.04. Btw, if we flash the latest JP5.1.2, could that make the jetson incompatible with the SD card containing the current image? Obviously we need to fix the problem, but we also need the sd card image to work as it is configured at the moment because there are people that need to work with it
Is the image used with Balena Etcher based on ext4? If so, then it would create an ext4 partition. However, if that content is corrupt, then it would create a corrupt ext4 partition. I am curious, when the system runs (it can be any of the Jetsons which use that same Balena Etcher source), what do you see from “df -H -T /”?
I couldn’t tell you if the QSPI of the prior release is compatible with the newest release, but the Orin software is the newest, and there are a lot of updates and changes. It would still be advisable, even if compatible, to flash the module itself when migrating. Maybe it won’t matter, but it is a moving target.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.