Hello,
I do have a problem setting the IPoIB interface MTU in VmWare 5.5 using the 1.8.2 drivers.
The adaptors i use are MCX354A-FCBT (dual-port IB/VPI adaptors - ConnectX3). They have been flashed with the latest firmware 2.30.8000 recently. The same issue applied when using previous firmware.
I have set the mtu_4k parameter to 1 for the mlx4_core module and restarted several time.
The issue I’m having is when setting the 4092 MTU value for the vSwitch who has the vmnic_ib0 adaptor as uplink, i’m getting the:
“Unable to set MTU to 4092 the following uplinks refused the MTU setting: vmnic_ib0” message.
Could you please explain if there are additional steps required for this? Am I missing something?
Thanks in advance…
Hi there,
Changes in the MTU size for an IPoIB pkey can be tricky. Its always possible to decrease, but increasing might not.
The first device connected will set the MTU size for the broadcast domain. i.e. subsequent connection of IPoIB interfaces will not be allowed to set an MTU size larger than the previous one.
So my suggestion is:
-disconnect all IPoIB interfaces attached to the partition (pkey interfaces)
- configure Subentmanager with the requested MTU size (like MTU=5 in partition.conf)
-restart SM
-make sure that the first interface added to the partition will use mtu size 4092
– example: in linux I would do as follows
echo 0x0001 > /sys/class/net/ib0/create_child
echo 4092 > /sys/class/net/ib0.8001/mtu
ifconfig ib0.8001 10.10.10.10/24 mtu 4092 up
and then check that the mtu really IS 4092 before adding the interfaces on the second node.
ref: RFC4391 Transmission of IP over InfiniBand (IPoIB)
5. Setting Up an IPoIB Link
The join operation is successful only if the Subnet Manager (SM)
determines that the joining node can support the MTU registered with
the broadcast group [RFC4392 Home | Internet-Draft Author Resources ] ensuring support for a common link MTU.
Hi!
Your procedure was correct!
And there is plus 1 step!
That’s a IB Partition configuration.
If you complete partition configuration restart SM then you can configura vSwitch MTU to 4092 bytes.
Try the procedure I think helped me also…
Disconnect all interfaces from the switch… Then connect a 4k configured interface to the switch (be sure it’s 4k)… Later connect the others… It should work… Also restart the SM when no interface is connected…
Good luck…
Thanks for that info… It might be this case… I’ll try this approach today…
One more question: can the required partition be the default one? FFFF in my case?
In my current config I did set the partitions.conf correctly using MTU=5, so that was included in my test.
I guess you can use default pkey ffff. although in my setup we’ve used another one, matching the IB-p_key value with a corresponding VLAN id on the switch.
Good luck!
Sorry that I am digging up an answered question, but I think I have the same problem with my setup.
I am running Vmware ESXi 5.1 with DDR IB interfaces, connected to a HP 4x DDR Switch, I have installed SM on the ESXi host (opensm) and I cant manage to set up the MTU = 4k
That’s the error, vmware is giving me :
Call “HostNetworkSystem.UpdateVirtualSwitch” for object “networkSystem” on ESXi “192.168.4.37” failed.
Operation failed, diagnostics report: Unable to set MTU to 4092 the following uplinks refused the MTU setting: vmnic_ib0
The mtu_4k is set to 1, the partitions.conf file is as follows : Default=0x7fff,ipoib,mtu=5:ALL=full;
The weird part is that I was able to configure it on one of the hosts, but can’t do it on the others.
Any ideas on how to make the other hosts use mtu = 4092 ?
Thanks a lot in advance!
P.S I am using the MLNX-OFED-ESX-1.8.1.0 version of the driver.
Hi!
Here is my PERSONAL LAB’s configuration.
You must configure your ESXi host, too!
- I think you miss the default configuration value of Mellanox vSphere OFED 1.8.2.
Here is sample script.
esxcli system module parameters set -m=mlx4_core -p=“mtu_4k=1 msi_x=1”
If you configure your Partition on your SM and install IB HCA driver for ESXi correctly,
there is a final step that you must run this script on your ESXi host’s shell (or SSH) console and reboot your host.
Then you can configure your vSwitch to MTU=4092.
- Maximum value isn’t 4096.
Good luck~!
