I have a Mellanox MSB-7890 externally managed switch and I have five nodes now that all have a Mellanox ConnectX-4 dual port 4x EDR 100 Gbps card (MCX456A-ECAT).
One of the nodes runs the subnet manager.
All of the nodes are running CentOS 7.6.1810 with ‘Infiniband Support’ package group installed.
When I go to run ibdiagnet, it shows:
I---------------------------------------------------
I- IPoIB Subnets Check
I---------------------------------------------------
I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
W- Suboptimal rate for group. Lowest member rate:40Gbps > group-rate:10Gbps
However, when I run iblinkinfo, this is what it shows:
CA: aes0 mlx5_0:
0x7cfe9003004431f8 1 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 5 "SwitchIB Mellanox Technol
ogies" ( )
CA: aes2 mlx5_0:
0x248a0703002b1ec6 4 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 3 "SwitchIB Mellanox Technol
ogies" ( )
CA: aes3 mlx5_0:
0x248a0703002b1eca 2 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 2 "SwitchIB Mellanox Technol
ogies" ( )
CA: aes4 mlx5_0:
0x248a0703002b1ece 6 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 1 "SwitchIB Mellanox Technol
ogies" ( )
Switch: 0xec0d9a0300224e70 SwitchIB Mellanox Technologies:
3 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 6 1 “aes4 mlx5_0” ( )
3 2 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 2 1 “aes3 mlx5_0” ( )
3 3 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 4 1 “aes2 mlx5_0” ( )
3 4 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 5 1 “aes1 mlx5_0” ( )
3 5 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 1 1 “aes0 mlx5_0” ( )
How do I set the subnet manager speed rate to be something higher than 10 Gbps?
I tried looking through the documentation and doing research online and it doesn’t appear that anybody else has encountered this before?
(I found this out while I was trying to find out what the MTU was on the switch. All of the cards are set to ‘connected mode’ with a MTU of 4092 (/etc/sysconfig/network-scripts/ifcfg-ib0).)
Your help is greatly appreciated.
Thank you.