Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

inbusiness.jhchoi · October 23, 2016, 2:36pm

Thank you for your reply.

I experienced infiniband iSER, SRP on OmniOS. Both protocol almost same performance and latency. But Mellanox don’t support on vSphere 6 and above.

I did downgrade to ESXi 5.5u3 latest patch and vCSA6u2 latest today.

I’ll wait to launch vSphere 6.5 then test ESXi 6.5 PVRDMA.

If this new ESXi 6.5 PVRDMA that co-work Mellanox and VMWare support network only, I’ll change major storage protocol.

Have a nice day!

Jaehoon Choi

mpogrebisky · December 18, 2016, 11:47pm

OK, 4 moths later, ESXi 6.5 released, and still no reply from Mellanox. As expected, previously working workarounds (uninstalling inbox drivers and installing 1.8.x.x) for SRP/iSER over IPoIB no longer work. I even attempted using 1.9.10.5 ETH drivers with my Connect-X3s and employing iSER, but, still, this didn’t work either. Which means I have to stick to ESXi 6.0, possibly, until I decide to EoL my Mellanox hardware altogether.

Mellanox: what’s your response to all this? Will there ever be a solution for RDMA storage for VMware, or you just decided it’s not worth the effort?

mpogrebisky · October 23, 2016, 9:07am

ZFS on Linux seems to be very stable for me.

Bear in mind that with LIO you will be able to use SRP only with CentOS inbox drivers. If you choose to install Mellanox OFED, you will have access only to iSER over LIO. The only way to have SRP with Mellanox OFED is using SCST.

I actually conducted some tests comparing VMware storage performance of SRP over LIO/Inbox, iSER over LIO/Inbox, iSER over LIO/OFED, SRP over SCST/OFED and iSER over SCST/OFED. In all tests SCST/OFED performed consistently better than ILO, with SRP over SCST/OFED being the best latency-wise and iSER over SCST/OFED bandwidth-wise.

inbusiness.jhchoi · December 20, 2016, 1:54pm

Hi!

ESXi 6.5 include RDMA support, but there is several bugs.

ESXi hypervisor embedded RDMA support that coworks VMware and Mellanox, but there are many advance will be required.

I think almost 2~3 years later they will support iSER and vRDMA properly.

Therefore ConnectX3,4,5,6 will be deprecated then new generation HCA will support it.

Mellanox always said that their product support everything, but only Linux support.

Jaehoon Choi

anvandare110 · January 13, 2017, 8:10am

Have any of you tried the “esxcli rdma iser add” command? I tried it on one host, it didn’t generate any output but this was logged in dmesg log:

2017-01-13T08:05:35.347Z cpu17:67870 opID=3f4390b7)World: 12230: VC opID esxcli-f2-cc96 maps to vmkernel opID 3f4390b7

2017-01-13T08:05:35.347Z cpu17:67870 opID=3f4390b7)Device: 1320: Registered device: 0x43044f391040 logical#vmkernel#com.vmware.iser0 com.vmware.iser (parent=0x1f4943044f3912b0)

I’m using native drivers on a clean esxi 6.5 install. ConnectX-3 10GbE nics (MT27520). My hope was that there are plans to add native iser support but might just be wishful thinking?

VMware vSphere 6.5 Documentation Library Online Documentation - vSphere Command-Line Interface Reference - VMware {code}

jason.cecchin · January 26, 2017, 2:34am

And in my usage scenario there is another part to this problem. I also need Windows connectivity to the same storage as ESXi.

Problem is with Server 2012 there is no SRP/iSer functionality at all because MS are focussing all their efforts onto SMB Direct (SMB3 over RDMA). (the problem with this is that it won’t talk to anything non MS which I guess is part of their plan also) There is however legacy SRP functionality for Server2008 so that solves the issue for the short term and also confirms the implementation of SRP is right solution for the right time.

There is however at the moment no iSer functionality for Server 2008 (and not likely to ever be because its too old) and none for Server 2012 (because MS haven’t done it and are not interested in doing it) - BUT (and its a big but) I believe there is work on an iSer client for Windows 2012 by a third party but I have no idea how its going.

erezsc · February 13, 2017, 2:25pm

Hi,

I’m sorry you understand it this way, Mellanox is actually dedicated very much to ESXi, and working closely with vmware on a day to day basis. The new native driver has limitations compared to the vmklinux driver, it simply does not allow us to support infrastructure that we could before such as IB and iSER. Vmware are aware of the drawbacks of the move from vmklinux, and we are working with the limitations at hand. As stated above we are trying to bypass the limitations currently set by offering SR-IOV solutions, while discussing with vmware future support for IB and iSER. I hope to bring news in the near future.

Erez.

inbusiness.jhchoi · January 16, 2017, 1:41pm

Hi!

I’m also have many questions about support iSER in ESXi environment.

Current RDMA support on ESXi is a half on VM guest OS - JUST SUPPORT LINUX GUEST ONLY - and some vRDMA functions conflict against Physical RDMA Hardware functions, some driver’s name space and some bug is critical.

I think also Mellanox and VMware focus on RDMA VM network for VM guest (vRDMA), not a RDMA Storage protocol now…

There is a difference between ESXi and KVM.

This difference cause a problem driver and storage protocol porting to ESXi.

ESXi hypervisor embedded drivers for VM guest, but native RDMA, vRDMA for VM guest now.

I’m also wait very long time that Mellanox support STABLE RDMA Storage protocol and SRIOV on ESXi environment.

But not now.

inbusiness.jhchoi · January 31, 2017, 4:00am

If you use vSphere OFED 1.8.2.4, 1.8.2.5, 1.8.3.0 on ESXi 6.0 with Solaris

COMSTAR SRP, iSER target then you will meet the ESXi PSOD.

ESXi 6.0 support Linux target only.