How to create IB partition and test it ?

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hi,any body did the test?

my environment ：

server1

ibstat

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 2

Firmware version: 2.11.500

Hardware version: 0

Node GUID: 0xf4521403007dc820

System image GUID: 0xf4521403007dc823

Port 1:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 10

LMC: 0

SM lid: 2

Capability mask: 0x02514868

Port GUID: 0xf4521403007dc821

Link layer: InfiniBand

Port 2:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 15

LMC: 0

SM lid: 56

Capability mask: 0x02514868

Port GUID: 0xf4521403007dc822

Link layer: InfiniBand

server 2

ibstat

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 2

Firmware version: 2.32.5180

Hardware version: 0

Node GUID: 0x248a070300f041f0

System image GUID: 0x248a070300f041f3

Port 1:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 25

LMC: 0

SM lid: 2

Capability mask: 0x02514868

Port GUID: 0x248a070300f041f1

Link layer: InfiniBand

Port 2:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 4

LMC: 0

SM lid: 56

Capability mask: 0x02514868

Port GUID: 0x248a070300f041f2

Link layer: InfiniBand

ib switch ：

switch-bbb978 [standalone: master] (config) # show ib partition

Default

PKey = 0x7FFF

defmember = full

ipoib = yes

members

GUID=‘ALL’ member=‘default’

under the default partition， ibping is ok .now I do not want to server1 can ping server2 。

i confiure like this on switch

switch-bbb978 [standalone: master] (config) # show ib partition

Default

PKey = 0x7FFF

defmember = full

ipoib = yes

members

GUID=‘ALL’ member=‘default’

test1

PKey = 0x0020

defmember = limited

ipoib = no

members

GUID=‘24:8a:07:03:00:f0:41:f1’ member=‘limited’

GUID=‘f4:52:14:03:00:7d:c8:21’ member=‘limited’

if i delete the default partition, other node can be effected.

if i keep the default partition,server 1 still can ibping server 2 ,how to set the priority ?

2

The fact that there is no default partition probably explains why ibping fails.

The default partition must be present. The way to isolate the nodes is to make them all limited members of this partition with the exception of the SM node which must be full member of that partition (0x7fff). Even in this configuration, ibping will fail as limited member cannot talk to another limited member.

Also, I don’t understand why limited member is being configured on partition 0x0009.

xu chen wrote:

thks.

i configure my switch like this ,

first remove default partition – no ib partition Default

then create partition like below

switch-bbb978 [standalone: master] (config) # show ib partition

test1

PKey = 0x0005

defmember = full

ipoib = yes

members

GUID=‘24:8a:07:03:00:dd:c9:01’ member=‘limited’

GUID=‘24:8a:07:03:00:f0:41:f1’ member=‘full’

test9

PKey = 0x0009

defmember = full

ipoib = yes

members

GUID=‘24:8a:07:03:00:dd:ba:21’ member=‘default’

GUID=‘f4:52:14:03:00:82:81:51’ member=‘default’

the server in each partition can ping each other with the child interface ib0.8005,but can not ping each other with ib0,and cannot ibping -L lid .

Is this configuration right?

3

Hi, partitions are IPoIB interfaces and do not limit IB communications between the nodes. You might try the following,

Unload ib_ipoib module ( you’ll have no ibX interfaces)

ibping between the nodes

Note: for some reason, you have two different subnet managers on the nodes (see sm lid in ibstat output). I would suggest explicitly specify interface/port when running ibping/ibserver or disconnect the other second port on each node.

4

thks

i got it。

5

thks.

i configure my switch like this ,

first remove default partition – no ib partition Default

then create partition like below

switch-bbb978 [standalone: master] (config) # show ib partition

test1

PKey = 0x0005

defmember = full

ipoib = yes

members

GUID=‘24:8a:07:03:00:dd:c9:01’ member=‘limited’

GUID=‘24:8a:07:03:00:f0:41:f1’ member=‘full’

test9

PKey = 0x0009

defmember = full

ipoib = yes

members

GUID=‘24:8a:07:03:00:dd:ba:21’ member=‘default’

GUID=‘f4:52:14:03:00:82:81:51’ member=‘default’

the server in each partition can ping each other with the child interface ib0.8005,but can not ping each other with ib0,and cannot ibping -L lid .

Is this configuration right?

6

Maybe this is terminology:

Rather than “partitions are IPoIB interfaces”, an IPoIB interface is on some partition and partitions are configured by SM.

7

From the ibstat output ( different sm lid on each port) seems lime each port is connected to a different IB fabric

8

Default partition is required and all nodes must be part of that partition but they need not be full members. SM/SA node must be full member of default partition. Then none of the nodes will be able to communicate with other nodes except SM/SA node.

You can then create additional partitions supporting IPoIB with groups of nodes as (full) members which can ping each other.