IB Switch IS5035 MTU Setting?

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

I am wondering if someone can help with this.

We have 2x Mellanox IS5035 Switches running SM.

I have set the MTU on all of our Mellanox ConnectX3 Adapters on our Windows 2012 and 2016 Servers to 4092.

We get the following error in the Windows Server 2016 Event log

“According to the configuration under the “Jumbo Packets” advanced property, the MTU configured for device Mellanox ConnectX-3 IPoIB Adapter is 4092. The effective MTU is the supplied value + 4 bytes (for the IPoIB header). This configuration exceeds the MTU reported by OpenSM, which is 2048. This inconsistency may result in communication failures. Please change the MTU of IPoIB or OpenSM, and restart the driver.”

In the IS5035 under Fabric MGMT > Partitions the Default MTU is set to Default.

Is this the correct place to change the SM MTU?

Will it take effect right away?

Will there be any interruption in traffic? These are production servers so I need to know the impact

Thanks!

Todd

2

All Mellanox embedded SM’s MTU is 2k(2048-4=2044).

Therefore you should change default partition’s MTU to 4k(4096-4092).

Best Regard,

Jae-Hoon Choi

3

Thanks.

Can you tell me if this will take effect immediately, or require a restart of the servers.

Will it cause an interruption of traffic?

Our IB is for traffic from Cluster nodes to SAN storage. I just want to know what the impact will be.

Thanks again!

Todd

4

The error should clear once the SM’s partition configuration (IpoIB MTU setting) is adjusted (to eliminate the inconsistency):

For example, to enable a 4096-byte IpoIB MTU on the Subnet Manager’s default partition, (assuming SM is running on a switch), perform the following commands below. If more than one switch is running SM, this change should be made on each switch running SM.

In MLNX-OS, to change this MTU setting, the path to this setting is:

In GUI:

IB SM Mgt tab

Partitions Tab

In the existing Default Partition,

IPoIB MTU can be changed from 2K to 4K.

No other settinngs need to be changed.

Apply changes, but be aware that this is an intrusive configuration change and will disrupt the cluster as the SM process gets restarted and the MTU changes are applied.