I have a trouble that I can't change Ethernet to Infiniband.

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I am trying to connect two machines with infiniband (MCX353A-QCBT) in Ubuntu 14.04. But I failed.

This is my state. (lspci, ifconfig, ibstat, ibnodes, /sbin/connectx_port_config, dmesg). Does anyone have any directions, links, or advice?

sangmuk@sclab1:~$ lspci | grep Mellanox

83:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

sangmuk@sclab1:~$ ifconfig

eth0 Link encap:Ethernet …

eth1 Link encap:Ethernet …

eth2 Link encap:Ethernet HWaddr 7c:fe:90:16:8b:50

inet6 addr: fe80::7efe:90ff:fe16:8b50/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:126402 errors:0 dropped:0 overruns:0 frame:0

TX packets:126585 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:23725053 (23.7 MB) TX bytes:23771526 (23.7 MB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:390193 errors:0 dropped:0 overruns:0 frame:0

TX packets:390193 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:26791107 (26.7 MB) TX bytes:26791107 (26.7 MB)

sangmuk@sclab1:~$ ibstat

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 1

Firmware version: 2.36.5000

Hardware version: 1

Node GUID: 0x7cfe900300168b50

System image GUID: 0x7cfe900300168b53

Port 1:

State: Active

Physical state: LinkUp

Rate: 10

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x0c010000

Port GUID: 0x7efe90fffe168b50

Link layer: Ethernet

sangmuk@sclab1:~$ ibnodes

ibwarn: [27593] mad_rpc_open_port: client_register for mgmt 1 failed

src/ibnetdisc.c:766; can’t open MAD port ((null):0)

/usr/sbin/ibnetdiscover: iberror: failed: discover failed

ibwarn: [27598] mad_rpc_open_port: client_register for mgmt 1 failed

src/ibnetdisc.c:766; can’t open MAD port ((null):0)

/usr/sbin/ibnetdiscover: iberror: failed: discover failed

sangmuk@sclab1:~$ ibhosts

ibwarn: [27635] mad_rpc_open_port: client_register for mgmt 1 failed

src/ibnetdisc.c:766; can’t open MAD port ((null):0)

/usr/sbin/ibnetdiscover: iberror: failed: discover failed

sangmuk@sclab1:~$ sudo /sbin/connectx_port_config

ConnectX PCI devices :

|----------------------------|

| 1 0000:83:00.0 |

|----------------------------|

Before port change:

eth

|----------------------------|

| Possible port modes: |

| 1: Infiniband |

| 2: Ethernet |

| 3: AutoSense |

|----------------------------|

Select mode for port 1 (1,2,3): 1

WARNING: Illegal port configuration attempted,

Please view dmesg for details.

sangmuk@sclab1:~$ dmesg

…

[338561.776021] mlx4_core 0000:83:00.0: Requested port type for port 1 is not supported on this HCA

2

Hi Sangmuk Jo,

Can you please send the output of:

“ibv_devinfo”

or

  1. mst start

  2. flint -d mlx4_0 q

3

Thanks,

Do you have MLNX_OFED installed?

4

Hi!

My “ibv_devinfo” and “flint”.

sangmuk@sclab1:~$ ibv_devinfo

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.36.5000

node_guid: 7cfe:9003:0016:8b50

sys_image_guid: 7cfe:9003:0016:8b53

vendor_id: 0x02c9

vendor_part_id: 4099

hw_ver: 0x1

board_id: MT_1060110018

phys_port_cnt: 1

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

sangmuk@sclab1:~$ flint -d mlx4_0 q

Image type: FS2

FW Version: 2.36.5000

FW Release Date: 26.1.2016

Product Version: 02.36.50.00

Rom Info: type=PXE version=3.4.718 devid=4099

Device ID: 4099

Description: Node Port1 Port2 Sys image

GUIDs: 7cfe900300168b50 7cfe900300168b51 7cfe900300168b52 7cfe900300168b53

MACs: 7cfe90168b50 7cfe90168b51

VSD:

PSID: MT_1060110018

5

Yes, I installed MLNX_OFED_LINUX-3.2-2.0.0.0-ubuntu14.04-x86_64.iso.

6

can you try using mlxconfig?

mlxconfig -d mlx4_0 s LINK_TYPE_P1=1

7

Thank you!

I changed to infiniband.

Thank you, Sangmuk