Switches have two links between each other (ports 1-1 and 2-2). Each server has one link to each switch (Ports 34-36).
Here is the output of net discover:
4036-SW1(utilities)# ibnetdiscover
Topology file: generated on Thu Jun 30 10:32:28 2016
Initiated from node 0008f10500203b28 port 0008f10500203b28
vendid=0x8f1
devid=0x5a5a
sysimgguid=0x8f10500109553
switchguid=0x8f10500109552(8f10500109552)
Switch 36 “S-0008f10500109552” # “Mellanox 4036 # 4036-SW2” enhanced port 0 lid 6 lmc 0
[1] “S-0008f10500203b28”[1] # “Mellanox 4036 # 4036-SW1” lid 1 4xQDR
[2] “S-0008f10500203b28”[2] # “Mellanox 4036 # 4036-SW1” lid 1 4xQDR
[34] "H-0002c903004e445a"1 # “IGA-S2D1” lid 2 4xQDR
[35] "H-0008f104039a3c1c"2 # “IGA-S2D2” lid 5 4xQDR
[36] "H-0008f104039a4e3c"2 # “IGA-S2D3” lid 8 4xQDR
vendid=0x8f1
devid=0x5a5a
sysimgguid=0x8f10500203b29
switchguid=0x8f10500203b28(8f10500203b28)
Switch 36 “S-0008f10500203b28” # “Mellanox 4036 # 4036-SW1” enhanced port 0 lid 1 lmc 0
[1] “S-0008f10500109552”[1] # “Mellanox 4036 # 4036-SW2” lid 6 4xQDR
[2] “S-0008f10500109552”[2] # “Mellanox 4036 # 4036-SW2” lid 6 4xQDR
[34] "H-0002c903004e445a"2 # “IGA-S2D1” lid 3 4xQDR
[35] "H-0008f104039a3c1c"1 # “IGA-S2D2” lid 4 4xQDR
[36] "H-0008f104039a4e3c"1 # “IGA-S2D3” lid 7 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x8f104039a4e3f
caguid=0x8f104039a4e3c
Ca 2 “H-0008f104039a4e3c” # “IGA-S2D3”
1 “S-0008f10500203b28”[36] # lid 7 lmc 0 “Mellanox 4036 # 4036-SW1” lid 1 4xQDR
2 “S-0008f10500109552”[36] # lid 8 lmc 0 “Mellanox 4036 # 4036-SW2” lid 6 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x8f104039a3c1f
caguid=0x8f104039a3c1c
Ca 2 “H-0008f104039a3c1c” # “IGA-S2D2”
1 “S-0008f10500203b28”[35] # lid 4 lmc 0 “Mellanox 4036 # 4036-SW1” lid 1 4xQDR
2 “S-0008f10500109552”[35] # lid 5 lmc 0 “Mellanox 4036 # 4036-SW2” lid 6 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903004e445d
caguid=0x2c903004e445a
Ca 2 “H-0002c903004e445a” # “IGA-S2D1”
1 “S-0008f10500109552”[34] # lid 2 lmc 0 “Mellanox 4036 # 4036-SW2” lid 6 4xQDR
2 “S-0008f10500203b28”[34] # lid 3 lmc 0 “Mellanox 4036 # 4036-SW1” lid 1 4xQDR
4036-SW1(utilities)#
Ping and data transfer works, but transfer speed is only about 2gb/sec which seems very low. Still trying to figure out if I have a switch/fabric issue or if the slow speed is just configuration. Server are Windows using the latest supported Mellanox driver for the Connectx-2 cards (4.80) and firmware (2.10.720). RDMA appears to be detecting fine on the hosts, so at least some of the config seems to be correct =/