Orin Nano UEFI firmware sometimes fails to detect NVME

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
,
1

Hi,

We got a report from users that sometimes booting stops in the EFI shell, and that happens because the NVME is not detected by the UEFI firmware. Executing map -r in the UEFI shell confirms the NVME was not detected.

This reportedly happens only sometimes, if power is pulled from the device, but it may also occur after a normal power off.

The device can be booted after a failed attempt if it is left powered off briefly and then powered back on. Then, the UEFI firmware will be able to detect it and boot normally, with no other intervention.

The problem happens with an off-the-shelf carrier board - not the Orin Nano devkit - and we are wondering if the NVME initialization / detection timeout can be configured or retried in the UEFI firmware. If that is possible, could you share a patch that we could test with?

Thank you

2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with the Jetson Orin Nano, where the booting process stops in the EFI shell due to the NVME not being detected by the UEFI firmware, is a known problem. This issue is not specific to the Orin Nano devkit, but rather a general issue with the UEFI firmware’s NVME detection.

The UEFI firmware has a limited timeout for detecting NVME devices, and if the device is not detected within this timeframe, the boot process will fail. This timeout is not configurable through the UEFI settings, but it can be modified through a patch to the UEFI firmware.

To address this issue, you can try the following:

  1. Increase the NVME detection timeout: You can try increasing the NVME detection timeout in the UEFI firmware. This can be done by modifying the NvmeDetectionTimeout variable in the UEFI firmware. However, this variable is not exposed through the UEFI settings, and modifying it would require a patch to the UEFI firmware.
  2. Retry NVME detection: Another approach is to retry NVME detection if the initial detection fails. This can be achieved by modifying the UEFI firmware to retry NVME detection after a brief delay.
  3. Use a different NVME driver: You can try using a different NVME driver, such as the nvme driver instead of the uefi_nvme driver. This may help improve NVME detection and resolve the issue.

Regarding the patch, I couldn’t find any publicly available patches that specifically address this issue. However, you can try contacting NVIDIA support or the Jetson Orin Nano community forum to see if anyone has developed a patch to address this issue.

In comparison to other Jetson modules, the Orin Nano is not the only one that experiences this issue. Other Jetson modules, such as the Jetson Xavier NX, have also been reported to have similar issues with NVME detection.

To mitigate this issue, you can try the following:

  • Use a high-quality NVME device that is compatible with the Jetson Orin Nano.
  • Ensure that the NVME device is properly seated and connected to the carrier board.
  • Avoid powering off the device abruptly, as this can cause the NVME device to not be detected by the UEFI firmware.
  • Try leaving the device powered off for a brief period before powering it back on, as this can help the UEFI firmware detect the NVME device.

In summary, the issue you’re experiencing with the Jetson Orin Nano is a known problem with the UEFI firmware’s NVME detection. While there are no publicly available patches to address this issue, you can try increasing the NVME detection timeout, retrying NVME detection, or using a different NVME driver to resolve the issue.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Some patches that we could test would be highly appreciated. Thank you

6

Hi AlexCo,

What’s the Jetpack version in use?

Could you help to clarify if the issue is specific to your custom carrier board?
If so, do you have custom design on PCIe or the power sequence different from the devkit

Please also use another NVMe(with different brand) to check if the issue still happens.

7

Hi KevinFFF - I’m the end user.

Carrier Board: AVerMedia D131L

NVMe: Innodisk M.2 4IE3 Series, MPN - DHM28-40GDH1KCADF(W)

8

Also, the Jetpack version used is Jetpack 6

9

Hi Alex and Peter,

Are you working together?

It is a custom carrier board designed by AverMedia.
Please help to clarify the items I mentioned in first response with your vendor.

Please share the result of cat /etc/nv_tegra_release on your board.

10

Hi @KevinFFF , the BSP is L4T 36.3.0:

# R36 (release), REVISION: 3.0, GCID: 36106755, BOARD: generic, EABI: aarch64, DATE: Thu Apr 25 03:14:05 UTC 2024
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

@peter.forrester may be able to clarify the items with the carrier board vendor.

Meanwhile, is there any quick change or patch that we could use for testing? For example the LLM response was about increasing the NvmeDetectionTimeout variable in UEFI and we’ve searched the edk2/edk2-nvidia sources but that name doesn’t appear to exist.

Thank you

11

Hi KevinFFF,

I have enquired with AVerMedia (Carrier Board vendor) and will reply ASAP.

12

Could you try modifying the following line in NvmExpress.h?

#define NVME_HC_ASYNC_TIMER  EFI_TIMER_PERIOD_MILLISECONDS (1)

We still need you to clarify the issue with your vendor first.

13

Hi KevinFFF,

Apologies for the delay in reply. AVerMedia say that the PCIe architecture is the same on the D131L as it is on the NVIDIA ON DevKit, and that the power sequencing follows the Jetson Nano Product Design Guide (DG-09502-001).

14

Could you try modifying the following line in NvmExpress.h?

#define NVME_HC_ASYNC_TIMER  EFI_TIMER_PERIOD_MILLISECONDS (1)

Sure, it seems 1 is the currently used value: edk2/MdeModulePkg/Bus/Pci/NvmExpressDxe/NvmExpress.h at r36.3.0-updates · NVIDIA/edk2 · GitHub

What value(s) do you recommend us to test with?

Thank you

15

Could you reproduce the similar issue on the devkit?

Please simply try 10, 50, 100 to check if they could help.

16

Could you also apply the overlay_mb1bct_36.x.tbz2 from Jetson Linux 36.3 | NVIDIA Developer to check if it could help for the current issue?

17

Hi KevinFFF,

No, we could not reproduce on the DevKit. We have enquired with AVerMedia (carrier board manufacturer) on this also.

Can you explain the purpose of these / how they might help? Thanks.

18

There’s the known boot issue happening on certain Orin Nano module.

If you can not reproduce it on the devkit, then it may be another issue specific to your custom carrier board.

19

The mb1 fix and increasing the NVME timeout to 1000 did not help, we’ll try with larger values

20

Okay, please also check what difference between the devkit and custom carrier board(D131L) may cause current issue as we don’t hit similar issue on the devkit.

22

According to the manufacturer there’s no difference between the D131L and the devkit in this respect.

We’ve flashed a debug enabled UEFI firmware and only when the NVME is not detected, the following log appears during boot. This is the only difference, apart from NVME not being listed by the UEFI firmware:

PCIe Controller-4 Link is DOWN

When it’s detected:

ListPciDevices: Segment: FF	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0xFFFF	Device ID:0x0000
ListPciDevices: Segment: 01	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229E
ListPciDevices: Segment: 01	 Bus: 0x01	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10EC	Device ID:0xC822
ListPciDevices: Segment: 04	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229C
ListPciDevices: Segment: 04	 Bus: 0x01	 Device: 0x00	 Function: 0x00	Vendor ID: 0x1BC0	Device ID:0x1602 <---
ListPciDevices: Segment: 07	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229A
ListPciDevices: Segment: 08	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229C
ListPciDevices: Segment: 08	 Bus: 0x01	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10EC	Device ID:0x8168

When it’s not detected:

ListPciDevices: Segment: FF	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0xFFFF	Device ID:0x0000
ListPciDevices: Segment: 01	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229E
ListPciDevices: Segment: 01	 Bus: 0x01	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10EC	Device ID:0xC822
ListPciDevices: Segment: 04	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229C
ListPciDevices: Segment: 07	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229A
ListPciDevices: Segment: 08	 Bus: 0x00	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10DE	Device ID:0x229C
ListPciDevices: Segment: 08	 Bus: 0x01	 Device: 0x00	 Function: 0x00	Vendor ID: 0x10EC	Device ID:0x8168
23

Do you mean the issue is caused from that the PCIe C4 is DOWN when the NVMe is not detected?