it will help you understand the routing algorithms and networking for InfiniBand.
This is very small network. I would check that all the ports are being used, if there are not so many flows - let’s say just one flow, only one port will be utilized - (same in Ethernet)…
Assuming that adaptive routing is not enabled, available paths will be statically balanced, between each pair of Local IDs (LIDs). If only one LID exists on each end of the 2 cables, only one cable will be used. If a server has 2 ports (2 LIDs), there will be two sets of paths to that server from every other LID in the subnet, and the routing algorithms will try to statically allocate half of the paths to each cable. But the existence of multiple available paths doesn’t mean they’ll be utilized. Load balancing among two ports on the same server is a function of the Upper Layer Protocol (IPoIB, SRP, etc.) IPoIB doesn’t do load balancing; MPI stacks do…
See some examples here as well regarding unused links.
Hmm, Interesting, I did notice that the Node GUID was different from the Port GUID’s
So as long as the target address is the node GUID the IB will load balance ?
As an example:
# ibstat CA 'mlx4_0' CA type: MT25418 Number of ports: 2 Firmware version: 2.3.0 Hardware version: a0 Node GUID: 0x0002c9030002fb04 !!!!!!!!!!!!!!!!!!! System image GUID: 0x0002c9030002fb07 Port 1: State: Active Physical state: LinkUp Rate: 20 Base lid: 2 LMC: 0 SM lid: 1 Capability mask: 0x02510868 Port GUID: 0x0002c9030002fb05 !!!!!!!!!!!!!!!!!!! Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510868 Port GUID: 0x0002c9030002fb06 !!!!!!!!!!!!!!!!!!!