Jetson Orin Nano – Multiple "Failed" Messages While Booting

Hello Team,

I am using the Jetson Orin Nano Developer Kit. When I power on the board normally (not in recovery mode), I see several “FAILED” messages appear during the boot process on the connected monitor.

I have not flashed the board recently, and I am using the default Jetson Linux provided.
HDMI output appears
Fan spins
But many services show FAILED status on boot
Examples of what I see:

  • Failed to start Hostname Service
  • Failed to start Login Service
  • Failed to start Load Kernel Modules

Could you please guide me on:

  1. What causes these “failed” boot messages?
  2. Is there a way to reset or reflash the board to fix them?
  3. Any recommended way to clean install JetPack if needed?
    I appreciate your help. Please let me know what logs or details are needed.

Thanks,
Pavan


Hi,

Do you use an HDMI adapter because our dev kit only supports DP?

Thanks

Hi,

Yes, I am using an HDMI to DisplayPort adapter because my monitor only supports HDMI.

Could you please confirm whether this type of adapter is compatible with the Jetson Orin Nano Developer Kit?

I’m facing an issue where the board powers on, but I see “FAILED” messages during boot, and there’s no display output on the monitor.

Could this be caused by the adapter or display compatibility?

I would really appreciate it if you could guide me step-by-step to resolve this issue, or suggest any alternative setup (such as using a native DisplayPort monitor or a specific adapter).

Thank you so much for your support.

Regards,
Pavan


once see the board.

Just a comment on one of those errors, the part of Load Kernel Modules error. Regardless of the reason for failing to load modules, this implies none of the drivers or hardware which have drivers in the form of a module will work. Maybe (or maybe not) things like host name service depend on that, e.g., if the network hardware used a module format driver, then host name service could not succeed.

The base kernel Image has its location for where to find modules hard wired into it at the time of compile. There are actually multiple kernels possible on most Jetsons, and perhaps it is loading something other than the one expected due to a filesystem issue. Is this a dev kit, booting to SD card? Or is this a third party carrier board? It changes some answers.

Regardless of the answer, it is likely that the Jetson will need to be flashed. You wouldn’t necessarily need to flash any SD card though. It’s hard to give better advice without knowing exactly whether this is a developer’s kit, or if it uses a third party carrier board, or if it boots to some alternative media (such as NVMe).

Bottom line though is that it cannot find some content on the filesystem, and this triggers a chain of failures.

Hi,

Thank you for the clarification.

Yes, I am using the Jetson Orin Nano Developer Kit, and it’s booting from an SD card. I am using an HDMI to DisplayPort adapter, as my monitor only supports HDMI.

During boot, I noticed several “FAILED” messages, and based on one of the errors — particularly the “Load Kernel Modules” failure — I understand that this might prevent drivers or hardware components from functioning correctly. I also read that this may be due to the system not finding expected content in the filesystem, which could trigger a chain of failures.

Could this possibly be caused by the adapter, or is it more likely a filesystem or flashing issue?

If flashing is necessary, I would appreciate your guidance on how to reflash the developer kit correctly using SDK Manager or any recommended method.

Please let me know the best way forward. Thank you so much for your support.

Best regards,
Pavan

The HDMI to DisplayPort adapter shouldn’t be a problem. Adapters can still fail for various reasons, but that part is usually not a problem. VGA adapters basically guarantee failure.

Have you performed any kind of kernel customization? There is an initrd which is actually the first loaded filesystem. The kernel comes from the SD card for an Orin Nano dev kit, but the first module load is then from the initrd which is also in “/boot”. If the initrd is not found, or if the kernel is not found, or if the modules found in the initrd are incorrect and not compiled against that kernel configuration, then I would expect this to fail. Summary:

  • Kernel Image is in “/boot”, and this loads.
  • Module load is expected to find compatible modules in “/lib/modules/$(uname -r)/kernel”.
    • That location exists on the SD card, but this is not the first location searched or loaded.
    • There is also a “/lib/modules/$(uname -r)/kernel” in the initial ramdisk. This loads first, and it looks like maybe it was never reached.
    • This initrd acts as a kind of adapter between first kernel load and actual filesystem load. It is a simplified o/s filesystem tree purely in RAM. The Image is not from the initrd, but what the kernel Image loads is from the initrd. Then there is a pivot root whereby the “/” of the filesystem is transparently switched from the “/” of the initrd and the SD card’s ext4 filesystem. Logs usually mention these events, but I don’t see them.
  • You really need a serial console boot log to see events and to know for sure. Serial console would show exact details which the video isn’t really able to show. There are some variations on hardware for serial console, but this should be useful:
    https://jetsonhacks.com/2019/04/19/jetson-nano-serial-console/

This is almost always a flashing issue. Flashing updates the QSPI memory on the module itself, and this has a lot to do with finding the initrd and kernel Image. The SD card can be 100% perfect, but if the module itself has not been flashed, you will get failures like this.

FYI, Jetsons don’t have a hardware BIOS, but they do have the equivalent in software. For a Nano style in a dev kit, there is no eMMC, and that content is stored in the QSPI memory of the module. During flash it isn’t just the operating system which gets flashed, but is also the QSPI. The QSPI basically gets the system to the point of running a bootloader, and eventually this loads the kernel Image and the initrd filesystem. A small subset of critical modules load from the initrd. Then there is a pivot root to put the SD card’s “/” as the root of the filesystem instead of the initrd’s “/”. This results in the rest of the modules (the ones on the SD card instead of the initial ramdisk) being able to load on demand.

What did you do for flash? Incidentally, L4T is what actually gets flashed, and this is just Ubuntu once NVIDIA content has been added; JetPack/SDK Manager are just tools to perform the flash. The two are normally paired together. You can see a list of L4T releases and compatibilities here:
https://developer.nvidia.com/linux-tegra

For Orin both L4T R35.x and R36.x are available (maybe also a new release when Jetson Thor comes out). Once you have flashed with R36.x then any of the SD card images from R36.x will work without flashing again. Similarly, if you have R35.x flashed, then any of the SD card images for R35.x will work without flashing again. You cannot mix an R35.x flash with an R36.x SD card (or vice versa).

Hi,

Thank you for your detailed explanation earlier.

I’m using the Jetson Orin Nano Developer Kit (8GB eMMC version) — so there is no SD card slot on this model. I haven’t made any kernel customizations. The board was working fine earlier (I had deployed face detection programs), but now it fails to boot with several “FAILED” errors during startup, including “failed to load kernel modules” and “cannot find /dev/mmcblk0p1”.

I’ve captured the full boot log using a USB-to-TTL serial cable, and I’ve attached the error output as a [jetson_orin_nano_developer_kit.txt|attachment](upload://3YYsgKTSIK1gt3hxaihvdeMcvir.txt) (117.4 KB) jetson_orin_nano_developer_kit.txt file here for your review.

Based on your explanation, I suspect that either:

  • The initrd or kernel modules are missing/corrupted, or
  • The QSPI memory might be misconfigured and needs reflashing.

I tried flashing the board using SDK Manager (putting the board into recovery mode by shorting the FC REC pin to GND), and while the USB connection was detected, the flash failed mid-process** with a disconnect. I followed the recommended power/USB reconnection sequence and even used different USB cables/ports, but still encountered the issue.

Could you please check the attached error log and advise me:

  1. If the issue is clearly from the initrd/kernel/modules.
  2. If there is any way to recover or reflash QSPI successfully in this situation.
  3. Whether I should use L4T R35.x or R36.x with this model.

Thanks so much again for your guidance.

Best regards,
Pavan

So far as I know the Orin Nano dev kits never have eMMC. I stress this because it changes a lot of details and tends to be quite different to flash for an eMMC model versus SD card (or external storage like NVMe). I’m thinking the 8 GB refers to RAM rather than eMMC storage. Any answer which actually had a flash using the wrong eMMC/SD/external NVMe storage detail would cause failure in some manner.

It looks like the log file attach didn’t take in your reply. Don’t know if the forum software did this or something else. This would be useful, but you might need to attach it again. If attaching fails, then what you can do is put two likes of three backquotes, like this, and the content pasted between (this adds scrollbars and quote formatting):

  • ``` (3 backquotes)
    …pasted content from long logs…
    ``` (3 more backquotes)

The backquote is normally the upper left corner of the keyboard, just below the escape key. This is ok for pasting long logs, but if you can fix the file attach of the log that would work too.

It is quite common to flash the QSPI again for non-eMMC models and having things start working again. You said there is no SD card slot, and I want to be very certain, is there no SD card slot (A) on the carrier board, and (B), is there no micro-SD slot mounted to the bottom side of the module and accessible at the edge of the module?

If the flash spec was wrong, then more likely that flash would not have started, but it can also fail in the middle. If you are using a VM for flash, then this is the most common failure case of stopping in the middle of flash with a disconnect. Jetsons purposely disconnect and reconnect during a flash, and a VM not correctly configured will not reacquire after the initial disconnect.

I would tend to recommend R36.x over R35.x if you don’t have specific requirements. If you have requirements for particular CUDA release version, then you’d want to check the specs of R35.x and R36.x for their release versions. Software compatibility is the main reason one would pick one or the other. R36.x also goes to mainline kernel and full UEFI, and so it is easier to work with if porting other content.

Note that if QSPI has R35.x content on it, then it won’t boot to R36.x content, and vice versa.

People do often flash just QSPI without changing the rootfs. Useful only if you have a rootfs you want to keep (I’d still recommend cloning the rootfs if you have something valuable, and then flashing QSPI).

Regarding the logs which didn’t show up, there is a strong possibility that even if an error does show up on the logs, then flashing would still be the answer. Just be certain you are flashing correctly for your model (eMMC versus SD versus NVMe, so on).

Dear Sir,

Thank you so much for your reply sir
I am writing to seek your kind support regarding an issue I am currently facing with my Jetson Orin Nano Developer Kit.

Due to 4 times power fluctuations, the board was turned on and off approximately four times. Then after i power on the board, it enters emergency mode during boot and does not proceed further.

I would like to humbly clarify the following points:

  • The board has an SD card slot, I am not using it currently. I apologize if there has been any confusion regarding this. I am not performing any operations related to the SD card at this stage. My only requirement is to get the board to boot normally again.
  • I have connected a UART cable (TX, RX and GND) to the board. I am able to observe the boot logs via the serial connection. I have attached the complete log file here for your kind reference, so that you can check the exact error message that is appearing during the boot process.

I have also attempted to flash the board using the NVIDIA SDK Manager, but unfortunately, the process failed. During the flashing process, the board gets disconnected, and I am unable to proceed further. I have attached screenshots of the error messages encountered during this process.

If it is convenient for you, I would be extremely grateful if we could schedule a brief session via Google Meet, Zoom, or Microsoft Teams, so I can show you the issue in real-time and share all relevant photos, error logs, and screenshots. .

Alternatively, if you could kindly share your email address or contact number, I would be happy to send all details to you directly.

As I am new to using the Jetson Orin Nano board, I would sincerely appreciate your expert guidance and support to help bring the board back to working condition.

Thank you very much for your time sir.

Regards
Pavan Jarupula
Email: jarupulapavan2002@gmail.com
Mobile: +91 99080 64619

Dear Sir,

I am encountering an error while uploading the log files. Please find the screenshot of the error message below:

Regards
Pavan

Dear Sir,

I am sharing the test file related to the issue I’m facing with the Jetson Orin Nano Developer Kit. Kindly check if you are able to access it?.
Please provide your email address so I can share all the necessary details directly.

jetson_orin_nano_developer_kit.txt (117.4 KB)

Regards
Pavan Jarupula
Email: jarupulapavan2002@gmail.com
Mobile: +91 99080 64619

Dear Sir,

Regards
Pavan

Dear Sir,

Regards
Pavan

Dear Sir,

Regards
Pavan

Regarding logs, they are plain text. One possibility is to split it into two text documents. I did get the new log though and it is useful.

I do not normally help outside of the forums, although private messages work. The problem there is that nobody else can benefit. I do think I see the problem though, so will add that down below. Some notes first…

Your external media is being found via the initrd. I don’t know yet if the filesystem recovery lost something important (I’m taking notes while I read, I haven’t read it all yet), but it did find and try to repair the ext4 partition on an NVMe:

[    8.015077] nvme nvme0: 6/0/0 default/read/poll queues
[    8.017297]  nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15
[    8.360975] EXT4-fs (nvme0n1p1): recovery complete
[    8.361907] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota .
[    8.369637] Rootfs mounted over PARTUUID=a8abf817-da8b-4459-8dae-63ad1c9933ed
[    8.376683] Switching from initrd to actual rootfs

Any flash does require (for your situation) an initrd flash, not the standard JetPack/SDKM (if you don’t need to flash don’t do this, but it is useful to know). There is a README type file here:
Linux_for_Tegra/tools/kernel_flash/

That README explains a command line to perform an initrd flash. That log line I showed though more or less says that everything worked correctly up to the initrd.

That initrd is a filesystem in compressed RAM, and pretty much all bootloaders and all o/s’s understand a compressed RAM filesystem (an initrd is an initial ramdisk). If you were to do something like boot to a filesystem type which the bootloader needs to be able to read, but is something odd, then you wouldn’t be able to boot; the initrd though will contain drivers and any other content (such as filesystem type knowledge) which the bootloader lacks. In the Jetson case this is not due to filesystem type, but due to the boot target. The boot chain normally expects you to boot to the SD card for the dev kit (mounted on the module itself). This particular initrd is redirecting boot to the NVMe. Linux is in fact running without error on this initrd, and any issue comes after a pivot root to the real filesystem on the repaired NVMe.

Some errors are unimportant, e.g., if you don’t have SElinux in use, then the SElinux label issue does not matter. My assumption though (and this could be wrong if there is some change I don’t know about) is that your system is not trying to enforce SElinux. I see no signs that SElinux has any meaning for your current issues.

I do see module or pseudo file failures. This is likely the problem (although just a symptom of something else related to filesystem corruption). It seems to be having problems with pseudo filesystems, i.e., things related to the kernel producing fake files in RAM looking like files which are really drivers…if the driver cannot load, then it cannot produce a pseudo file. Some of this includes boot related content, like the EFI (boot is out of EFI already, but EFI provides data or context to the running system; this could be on QSPI, but I’m thinking this is on the NVMe…maybe not, this isn’t certain; some content might not be used and might not matter). The bottom line is that anything not integrated into the kernel itself does not seem to be loading.

Basically, services needed for normal startup fail and it drops into a recovery shell:

Press Enter for maintenance
(or press Control-D to continue):

  • First, is this L4T R35.x or is it R36.x? This changes a lot, and you don’t want to mix instructions or software from one with the other.
  • Is there anything on the NVMe you need to save? If so, do you have a second NVMe to try or do you have a Linux system with enough space on it that you can copy the entire partition to it and not run out of space? The NVMe, when placed on another computer, can easily be cloned and saved in ways which work to save data or use it with flashes.
  • Do visit the Linux_for_Tegra/tools/kernel_flash/ location and examine the README file. There is an initrd flash script for use with the NVMe flash; other flash methods might appear to work, but won’t actually boot correctly for your situation.
  • If you don’t care about the NVMe (this is where the corruption appears to be), then you can just initrd flash. If you care about the NVMe, I can tell you how to save as much as possible before flashing again.

Dear Sir,

Thank you very much for your detailed explanation and continued support.

I have reviewed the information provided, and I would like to confirm that I do not need to back up any data from the NVMe. My goal is to fully restore my Jetson Orin board to its original factory condition, exactly as it was when I first purchased it with default settings and the default Linux image.

I understand that in my case an initrd-based flash is required and I kindly request your step-by-step guidance to perform this process correctly.

Thank you once again for your valuable time and guidance.

Best regards,
Pavan

The README file has the actual command listed in it if you want to boot normally to any device other than the SD card which is mounted directly on the module (if the SD card is on the carrier board, then this is not a developer’s kit, and would instead be a commercial module on a third party carrier board). Otherwise you’d just use normal flash software command line “sdkmanager” to start JetPack (and uncheck everything except flash; you could later add optional content like CUDA by unchecking everything except the software, having no flash, and using the Jetson fully booted instead of in recovery mode).

Some hints along the way:

  • If you are in “Linux_for_Tegra/” of the host PC, and you list these files:
    ls jetson*.conf
    …then remove the “.conf”, then these are all flash targets. The same is true for flash targets named by an initrd flash on command line. The name will basically describe a module and a carrier board. In your case you’re interested in a Jetson Orin Nano dev kit.
  • The Jetson is in recovery mode regardless of whether a flash is initrd or normal sdkmanager or other command line flash. A flashed Jetson automatically reboots when flash completes, and this step is not usually obvious, but it means that when you get to first boot account setup the Jetson is actually fully booted (this is when ssh is used to install extra software like CUDA if it is checked; uncheck and do this later if you want the simplest case).
  • The flash target for a regular flash on an Orin Nano dev kit is the SD card, and not an initrd flash to NVMe. For that the target partition is mmcblk1p1 (the eMMC partition for a non-SD card is mmcblk0p1…note blk0 versus blk1). This is just the first partition of that device. The target for an initrd flash of an NVMe would be nvme0n1p1 (this is the first partition of the NVMe).
  • Between naming the target and naming the flash partition there isn’t a lot more to do other than being in recovery mode with the correct USB connected (and simplifying if you use JetPack/SDKM by unchecking everything except flash).
  • There is an initrd flash helper script you can use which isn’t just for initrd flash, and it is listed in the README file: tools/l4t_flash_prerequisites.sh. Note how everything is listed as a subdirectory to the “Linux_for_Tegra/” directory.
  • If you are having trouble with first boot account setup, then there is another utility script (one you would not use for flashing a commercial product going to an end customer, but quite useful if you are flashing only for yourself):
    tools/l4t_create_default_user.sh
    You can use that to set up your first boot account before you even flash. Every flash, from then on, using that flash software, will have your first boot account and password setup completed and you won’t need to set that up. Especially useful if you can’t get first boot setup due to something like a video issue. Even ssh would work at that point with that login name/pass.
  • I will emphasize that you don’t need initrd flash if you are flashing the default SD card of an Orin Nano dev kit. As soon as you flash an SSD, NVMe, or any USB device, then you have to go to the initrd flash.
  • If you have problems, or even if you just want to document, then you can save any command line flash to a log as you perform the flash. Just append " 2>&1 | log_flash.txt" to the end, and you’ll get the log of flash in file log_flash.txt. Example:
    sudo ./tools/kernel_flash/l4t_initrd_flash.sh <board-name> <rootdev> 2>&1 | tee log_flash.txt

If you have an issue, just post again for information. If you have a command line flash and attach a log to the forum, then it is a lot easier to answer.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.