BMC does not finish initializing

Robotics & Edge Computing IGX Systems IGX Thor Support
1


Dear NVIDIA team,

During the setup of IGX Thor, I get the issue that BMC is not finishing the initialization.
Setup is being done following NVIDIA Base OS installation guide: Installing Software on IGX Thor Developer Kit — NVIDIA IGX Developer Guide

Symptoms:

  • BMC web UI is accessible.
  • BMC continuously displays “BMC is still initializing. Use at your own risk.”
  • System Inventory reports 0 processors and 0 GiB memory.
  • Secure Boot status does not load.
  • Power Operations page sometimes remains stuck showing “There are no options to display while a power operation is in progress.” when the device is turned off from BMC, even though the status changes to “off”.

Host status:

  • Thor powers on successfully.
  • UEFI detects CPU (~2.60 GHz) and RAM correctly.
  • System boots into UEFI shell.
  • The system boots into the UEFI shell, and the Boot Manager currently has no OS boot entries.

Troubleshooting done so far:

Diagnostic findings

  • BMC firmware: IGX-BMC-v5.4

  • SMCU firmware: NFW 2.00.02 (previously shown; now version not visible anymore [-])

  • NIC_CX7_15 firmware: 28.46.1006

  • Boot Firmware (QSPI): 38.5.0-gcid-44254405 (on UEFI screen) [Note: this is the only value that differs compared to the required version 38.5.0-gcid-44447062 from the user -guide for installation]

  • failed-services.log reports:

    • system-guid.service - failed
    • delete-hi-user.service - failed
    • proc-fs-nfsd.mount - failed

  • An earlier journal log also showed:

    • mctp-pcie-ctrl: Segmentation fault (core dumped)

      In some runs also the mctp-pcie-ctrl.service shows up as failed.

Also, there are two screenshots attached of the BMC Inventory and Firmware pages that reflect the issues described above. Please let me know if anything else is required apart from this information.

Looking forward to your support.

Best regards,
Dejana

Inventory
Inventory2622×1866 322 KB

Firmware
Firmware2630×1728 302 KB

3

Hi dejana.antic,

Please run the following commands to check the if BMC is ready.

[Host]
USER=root
PASS=<bmc password>
curl -sku "$USER:$PASS "https://192.168.1.110/redfish/v1/Managers/IGX_BMC_0" | jq '.Status.State, .UptimeSeconds'

You may also run journalctl -f in BMC to check the logs.

Have you tried reflashing the BMC firmware to check if it helps?

4

Hi,
I ran the requested Redfish query. The Manager reports:

Status.State = "Starting"
UptimeSeconds = null

I also checked the BMC logs using journalctl -f. The following message is repeatedly printed every 5 seconds:

[ERROR managers.hpp:3598] Failed to open OTP provisioning status file

I investigated the failed services further.

  • system-guid.service fails with:
Failed to set property UUID on interface xyz.openbmc_project.Common.UUID:

Unknown object '/xyz/openbmc_project/Common/UUID'.
  • mctp-pcie-ctrl.service fails with:
ExecStart=/usr/bin/mctp-pcie-ctrl

code=exited, status=139

Start request repeated too quickly.

Failed to start MCTP PCIe control daemon.

Regarding the BMC firmware update, I have not reflashed the BMC yet.

I checked the board part number (699-13809-0300-100), which, according to the firmware update documentation, corresponds to the Non-ERoT platform based on: Update Your BMC Firmware — NVIDIA IGX - Board Management Controller (BMC) . Please correct me if I am wrong.

Given the current state of the BMC (Status.State = “Starting”), the mctp-pcie-ctrl.service crash, and the repeated OTP provisioning status error, what would be the recommended next step?

Should I proceed with updating the BMC using the igx-bmc-apfw-non_erot.fwpkg package through the BMC Web UI, or is there another recovery or diagnostic step that should be performed first?

Thanks!

5

In my case, Status.State reports "Enabled".

Please proceed to updating BMC through BMC WebUI.
Or you can download BMC firmware from https://developer.nvidia.com/downloads/igx/v2.0/bmc_fw_R38.5.0_aarch64.tbz2 and update it by the script.

6

Hi,

I updated the BMC using the igx-bmc-apfw-non_erot.fwpkg package through the BMC Web UI as recommended.

The firmware update completed successfully. From the BMC logs:

Activating PLDM firmware update package
Updating bmc...
Finished Update mtd partition.
Firmware update time: 153071 ms
Activating PLDM firmware update package - BMC reboots are re-enabled.

I then performed Operations → Reset BMC → GracefulRestart.

Unfortunately, after the restart the behavior is unchanged.

The Redfish Manager still reports:

Status.State = "Starting"
UptimeSeconds = null

The Web UI still displays: “BMC is still initializing. Use at your own risk.”

mctp-pcie-ctrl.service is still failing with the same error after the firmware update. I also continue to see:

[ERROR managers.hpp:3598] Failed to open OTP provisioning status file

One additional observation is that after every BMC restart, the logs start from Jul 02 (although the time itself progresses normally).

Since the BMC firmware update completed successfully but the behavior is unchanged, what would you recommend as the next recovery or diagnostic step?

7

Could you check if updating the BMC through the script may help for your case?

Have you also tried to reset the BMC after update?