Bluefield-3 no access to IPMI device

Infrastructure & Networking BlueField
1

Hello.
On previous BFB-Bundles installations on our BF-3 we could obtain the BMC interface IP by executing (on ARM OS):

ipmitool lan print 1

We have 2x NVIDIA BlueField-3 B3210E. Both error out on this command with:

could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

We used to set the static IP of the BMC interface with:

ipmitool lan set 1 ipsrc static
ipmitool ln set 1 ipaddr
ipmitool lan set 1 netmask
ipmitool lan set 1 defgw ipaddr

We do not have access to the DHCP server. Without IPMI or knowing the BMC IP, we cannot switch to static IP, cannot SSH, cannot execute Redfish commands on the BMC.

As this is reproducible on at least 3 of our DPUs, when will NVIDIA fix this?

We set-up the DPUs with the latest BFB:

BlueField SW bundle supporting BlueField-3 & BlueField-2, including DOCA 3.1.0, DPU-OS Ubuntu 22.04, BSP 4.12.0.13720, NIC-FW BF2 24.46.1006, BF3 32.46.1006, BF3 BMC-FW 25.07, BF3 BMC-eROT 00.02.0195.0000, BF2 BMC-FW 25.07, BF2 BMC-eROT 04.0f

To summarize:

  • How can we get the DHCP IP of the BMC?
  • How can we set a static IP on the BMC, if no DHCP-enabled network is available? (funny use case, isnt it - wow!)
2

Hi

Your BF3 is DPU or NIC mode ?

You have DHCP server ?

Your BMC port is connected to switch ?

Thanks

3

Hi @tuanama

DHCP server is running inside the network where OOB interface is connected to.

DPU ARM OS gets IP via DHCP.

BMC most likely also gets IP via DHCP. We have NO ACCESS to the DHCP. But thats not the problem.

Problem is that ipmitool is not working (!) anymore on the Bluefield ARM OS in order to get the BMC IP.

With ipmitool and previous Firmware we could get the BMC IP that the BMC got from the DHCP server.

So, without ACCESS to the DHCP server, how is it possible to get the BMC IP and access it? (Without fancy USB cables etc…)

4

Hi,

I use this format for get or set in last version BSP:

root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost sdr info
SDR Version : 0x51
Record Count : 35
Free Space : > 64Kb - 2 bytes
Most recent Addition : 11/10/2020 23:25:38
Most recent Erase :
SDR overflow : no
SDR Repository Update Support : unspecified
Delete SDR supported : no
Partial Add SDR supported : no
Reserve SDR repository supported : yes
SDR Repository Alloc info supported : no
root@localhost:/home/ubuntu#

root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan print
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : NONE MD2 MD5 PASSWORD
: User : NONE MD2 MD5 PASSWORD
: Operator : NONE MD2 MD5 PASSWORD
: Admin : NONE MD2 MD5 PASSWORD
: OEM :
IP Address Source : Unspecified
IP Address : 0.0.0.0
Subnet Mask : 0.0.0.0
MAC Address : 00:00:00:00:00:00
SNMP Community String : public
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
Cipher Suite Priv Max : aaaaaaaaaaaaaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : Not Available
root@localhost:/home/ubuntu#

run
root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan set 1 ipsrc dhcp

root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : NONE MD2 MD5 PASSWORD
: User : NONE MD2 MD5 PASSWORD
: Operator : NONE MD2 MD5 PASSWORD
: Admin : NONE MD2 MD5 PASSWORD
: OEM :
IP Address Source : DHCP Address
IP Address : 0.0.0.0
Subnet Mask : 0.0.0.0
MAC Address : 00:00:00:00:00:00
SNMP Community String : public
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
Cipher Suite Priv Max : aaaaaaaaaaaaaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : Not Available

Check the documentation BlueField BSP v4.12.0

5

Hi again and thanks for clarification:

I read so many docs and still the BMC is not configuring itself either via DHCP or via Static IP.

This seems to be a firmware bug, as the BMC cannot be used in the current state.

@NVIDIA: Please respond!

What I tried (all from DPUs Arm Ubuntu 22.04 OS):

  • Set static IP via ipmitool → Static IP not pingable (see my walkthrough below)
  • Reflashed complete BFB image → same problems like before
root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost sdr info
SDR Version                         : 0x51
Record Count                        : 35
Free Space                          : > 64Kb - 2 bytes
Most recent Addition                : 11/10/2020 23:25:38
Most recent Erase                   :
SDR overflow                        : no
SDR Repository Update Support       : unspecified
Delete SDR supported                : no
Partial Add SDR supported           : no
Reserve SDR repository supported    : yes
SDR Repository Alloc info supported : no

root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan set 1 ipsrc static

root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan set 1 ipaddr 10.10.140.196
Setting LAN IP Address to 10.10.140.196

root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan set 1 netmask 255.255.0.0
Setting LAN Subnet Mask to 255.255.0.0


root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan set 1 defgw ipaddr 10.10.0.1
Setting LAN Default Gateway IP to 10.10.0.1


root@localhost:/home/ubuntu# ipmitool -U ADMIN -P ADMIN -p 9001 -H localhost lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD
Auth Type Enable        : Callback : NONE MD2 MD5 PASSWORD
: User     : NONE MD2 MD5 PASSWORD
: Operator : NONE MD2 MD5 PASSWORD
: Admin    : NONE MD2 MD5 PASSWORD
: OEM      :
IP Address Source       : Static Address
IP Address              : 10.10.140.196
Subnet Mask             : 255.255.0.0
MAC Address             : 00:00:00:00:00:00
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
Default Gateway IP      : 10.10.0.1
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
Cipher Suite Priv Max   : aaaaaaaaaaaaaaa
:     X=Cipher Suite Unused
:     c=CALLBACK
:     u=USER
:     o=OPERATOR
:     a=ADMIN
:     O=OEM
Bad Password Threshold  : Not Available
6

NVIDIA???

7

-Is the BMC fully/booted

Not fully/booted

[root@localhost ~]# i2cdetect -y 1

0 1 2 3 4 5 6 7 8 9 a b c d e f

00: – – – – – – – – – – – – –

10: – – – – – – – – – – – – – – – –

20: – – – – – – – – – – – – – – – –

30: UU – – – – – – – – – – – – – – –

40: – – – – – – – – – – – – – – – –

50: – – – – – – – – – – – – – – – –

60: – – – – – – – – – – – – – – – –

70: – – – – – – – –

Fully/booted

[root@localhost ~]# i2cdetect -y 1

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

00: – – – – – – – –

10: 10 UU – – – – – – – – – – – – – –

20: – – – – – – – – – – – – – – – –

30: – – – – – – – – – – – – – – – –

40: – – – – – – – – – – – – – – – –

50: – – – – – – – – – – – – – – – –

60: – – – – – – – – – – – – – – – –

70: – – – – – – – –

-Are the modules loaded

modinfo ipmi_si

lsmod |grep ipmi

ipmi_devintf 24576 0

ipmi_msghandler 81920 2 ipmi_devintf,ipmb_host

lsmod | grep ipmb_host

ipmb_host 24576 0

ipmi_msghandler 81920 2 ipmi_devintf,ipmb_host

-Check dmesg for errors

dmesg | grep -i ipmi

-Power cycle the server and check on the dpu /sys/bus/platform/devices/ à ipmi_bmc.0, ls /dev/ipmi0

-Push image again and confirm BMC is aligned to version 25.07 (Upgrade BMC if not aligned), you will need to have the bf.cfg in place ie:

cat /var/tmp/bf.cfg

BMC_USER=“root”

BMC_PASSWORD=“” à Default 0penBmc

UPDATE_BMC_FW=“yes”

UPDATE_CEC_FW=“yes”

- Check your DHCP server for leases from the BMC MAC (labelled on the BF3 board)

-ipmitool is still the tool to use on BF3 & latest image 3.1/ BMC 25.07-6 (validated & no issue)

-As a ref https://docs.nvidia.com/networking/display/nvidia-bluefield-bmc-software-v25-07.07.pdf