DGX-1V 32GB: [Mellanox CX-5 IB card] It takes long time(7 minutes and 30 seconds) to power on the system to POST screen with installed CX-5 IB cards.

1

It takes an unusually long time (around 7.5 minutes) after powering on a DGX-1V 32GB server to boot into the operating system (Ubuntu 16.04 in this case). We do not observe this long of a delay on boot with the CX4 HCAs with the Option ROM enabled. The boot time on the CX5 HCAs is improved and equal to that of CX4 HCAs by disabling the Option ROM on the CX5 HCA with all else being the same.

[System Configuration]

SBIOS: S2W_3A04

BMC: 3.20.30

CX-5 IB cards: Mellanox ConnetX-5 EDR + 100GbE(Model No: CX556A)

[Reproduced steps]

  1. Replace existing “CX-4 IB cards” x4 with “new CX-5 IB cards” x4 on DGX-1V 32GB server.

  2. Power on the system.

[Observation]

It takes long time(7 minutes and 30 seconds) to power on the system to POST screen with installed CX-5 IB cards.

[Comparison]

DGX-1V + CX-4 IB card x4: It takes 1 minute and 27 seconds to power on the system to POST screen.