MTU Size on SX6036

Infrastructure & Networking Switches and Gateways
#1

I have an SX6036 and am using it between 2 windows 2016 servers as an Oracle RAC interconnect. I am having problems that might be related to the MTU size. I’m not 100% sure.

Windows Event Viewer shows this error:

“According to the configuration under the “Jumbo Packets” advanced property, the MTU configured for device Mellanox ConnectX-3 IPoIB Adapter #2 is 4092. The effective MTU is the supplied value + 4 bytes (for the IPoIB header). This configuration exceeds the MTU reported by OpenSM, which is 2048. This inconsistency may result in communication failures. Please change the MTU of IPoIB or OpenSM, and restart the driver.”

So when login to the SX6036 and set the MTU to 2048, I get this error:

“Error - valid MTU values for ipoib are 252, 508, 1020, 2044, 4092”

What’s the solution for this? Can the network adapter be changed? Not sure what to do…

#2

I think I figured it out. I set the Jumbo Packet size (in Properties) to 2044 and in the switch to 2044 and the error message went away in the Event Viewer.

I’m not yet sure if this has solved the problems.

#3

Hello McGinity,

Thank you for posting your inquiry on the NVIDIA Networking Community.

Based on your update, your were able to align the MTU on the node on the default IPoIB MTU from the switch.

If you want to increase the network performance, you can also change the IPoIB MTU on the switch to 4K and align the node to 4K.

You can use following CLI syntax on the switch to configure the default IPoIB MTU to 4K:

ib-sw [standalone: master] > enable

ib-sw [standalone: master] # configure terminal

ib-sw [standalone: master] (config) # ib partition Default mtu 4K force

ib-sw [standalone: master] (config) # configuration write

ib-sw [standalone: master] (config) # exit

Thank you and regards,

~NVIDIA Networking Technical Support

#4

Yes, thanks for checking in. I got the MTU set and I also found the Mellanox WinMFT utility which updated the drivers. I am building a new Oracle RAC installation and chose IPoIB for the interconnect. I set this up on another server many years ago and it has been working nicely. In that installation, my switch did not run the subnet manager so I had to install it on the servers. This time, the switch has it.

In that installation, I disabled the “Large Send Offload V2” for IPv4/6 for some reason. So I did the same in this new build. Also the in the Tunning Tab, I set it “Default”.

Things seem to be working.

#5

I’m not sure if things are 100% ok. The MTU in the swtich and cards all match. I’m not sure if Oracle needs a partiular MTU size or the issues I’m seeing are unrelated. Looking at Oracle’s notes, they suggest an MTU size of 9000. The SX6036 cannot support this. If I turn off the Subnet Manager in the swtich and use the version which installs as a Windows Service can it support 9000?

#6

I have another question. Right now ib0 State MTU is 2044. When I try to set to 4092, it errors saying:

“Error setting MTU for interface ib0: RTNETLINK answers: Invalid argument”

I thought maybe it was because it needed updating so it is now update to PC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc.

#7

I just clicked the “Expand Post” link and read your full answer. (I missed that link. )

After running:

ib-sw [standalone: master] > enable

ib-sw [standalone: master] # configure terminal

ib-sw [standalone: master] (config) # ib partition Default mtu 4K force

ib-sw [standalone: master] (config) # configuration write

ib-sw [standalone: master] (config) # exit

ib0 the MTU is now in 4092. I ran it via the web API in “System - Configuration - Execute CLI commands”

I guess there is something not working when just doing it from “System-Interfaces”. Anyway, all good. I also made sure the 2 ports in “Ports” are set to 4k. And in the Windows, Adapter’s “Jumbo Packet” (in Properties - Configure - Advanced) is set to 4092.

It looks like everything is working. Mellanox Infiniband is awesome!