Linux VM communication on IPoIB Network

I was tried local private network from Ethernet to IPoIB.

But there is some problem occurred.

I have virtual Untangle VM for DHCP service and control separated private network.

Windows VM works greate on IPoIB vSwitched network like Ethernet vSwitched network.

But Linux VM wasn’t.

Exactly speaking Linux VMs between separate ESXi hosts.

Linux VM on same ESXi host works with DHCP service, but separate ESXi host environment wasn’t.

(My vUntangle VM on Host01, DHCP client Linux VM on Host02)

Manually configured Linux VM on separate ESXi host works good everytime.

Only problem is interaction with DHCP service.

Is there any suggestions?

IB configuration is here.

4036 QDR switch configured MTU=5 and partition properly in embedded SM.

All ESXi host was configured mtu_4k=1

All HCA is ConnectX-2 with firmware 2.10.0720.

I found a real truth of IPoIB definition last day.

There is a difference between IPoIB and Ethernet protocols.

Basic solution is eIPoIB support in vSphere environment.

I’ll waitng next vSphere OFED release several years later.

I think IB on vSphere environment was not solid and flexible like Ethernet with DCB or iWARP.


There is Funny situations.

Windows VM DHCP Server to Windows DHCP client VM → Success!

Windows VM DHCP Server to Linux DHCP Client VM → Success!

Linux VM DHCP Server to Windows DHCP Client VM → Success!

Linux VM DHCP Server to Linux DHCP Client VM → Fail!

All test was proceeded in seperated ESXi 5.5 hosts and 4k IPoIB environment.

That’s a very curious…

I found some information about this problem came from IPoIB features.

ESXi can’t support eIPoIB interface currently. I’ll waiting support it…:)