Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

This is mostly exactly the current question I am trying to answer for myself.

We have a few ESXi hosts currently on v5.5 with dual port VPI adapters.

We want to run SRP/iSER on one port back to our SAN and we want to run IPoIB on the other port for vMotion communications.

We don’t want to go to Ethernet based adapters and/or have to buy managed switches to achieve this.

Ultimately we don’t want to implement this and then find it won’t run of v6.x either so forwards compatibility is important also.

Section 4.3 of this article Errors | VMware Flings suggests that in-box ESXi drivers are RDMA-aware, which you can also conclude from the very fact one of the modules related to these drivers is called nlmx4(5)_rdma. That doesn’t mean though that RDMA is actually going to be effectively used for host-to-storage access. For iSCSI, that would require implementing iSER layer, which the in-box driver clearly doesn’t have (after all, it’s explicitly called “Software iSCSI Adapter” and it’s like 100 years old). And, if VMware NFS could utilise RDMA, I’m pretty sure they’d already told us about it.

So, someone has to do some work here, either Mellanox or VMware or both. But they’re not eager to tell us anything, as you can clearly see from 1/2 year lack of response to this thread…

Hi!

I saw a some link and VMworld 2016 USA INF 8469 iSCSI/iSER Hardware SAN Performance over the Converged Data Center on Youtube that was disappear on now…:(

This video show me a some future direction to iSER support on ESXi in future and presenter were two people from VMware & Mellanox.

iSCSI Adapter number rule was change on ESXi 6.5 like vmhba64.

I guess some plan was started and testing process also on going that to support Native iSER initiator support on ESXi 6.5 + Update @ in 2017 or next years.

Just stand!

Jaehoon Choi

To Erez

SR-IOV was aimed to support low latency network on GuestOS.

SR-IOV vNIC bypass hypervisor kernel then direct access to physical switch from VM.

Not for Storage protocol orginally.

Why do you think SR-IOV is solution for this old aged ESXi driver support problems?

I saw a multiple IPoIB vNIC, SRP initiator, Ethernet Support with single HCA port on vSphere 4.x.

These features reduced so many complexities in cabling, power consumption and etc.

AND …

Virtualizaton has a very big feature that hardware driver independent platform to GuestOS and Administrator!

Do you want to break away this big feature from virtualization world?

IPoIB, iSER, SRP on GuestOS?

Do you want to make so many Guest OS’s drivers and functions?

How?

You must make a new subnet manager for both physical & virtualization QPs, GUIDs, etc then equip it on a new managed switches & HCAs then you’ll be say it!

You must purchase a new ConnectX-7 switches & HCAs then you also wait to launch new driver for ESXi after several thousands years!!!

I’m also think to migration Solaris SRP Target to Linux LIO Target.

But I have a question about stability of ZFS on Linux.

What do you think about it?

Best Regards.

Jaehoon Choi

I heard launching about new vSphere 6.5 yesterday.

I’m also heard that Mellanox will drop a support Infiniband iSER on ConnectX-3 and belows from some source that I can’t remember exactly now.

I give up using infiniband then change all of SX6036G switch’s operation mode to Ethernet.

All ULPs were unstable on vSphere 6.0 Update 2 even vSphere 6.0 OFED 2.4.0, too.

Mellanox switch systems show me a good performance and low power consumption, but also show me a low satisfaction with CLI configuration and functions unlikely Dell, Arista and etc’s ones…:(

I will wait to public distribution of vSphere 6.5 then I will make a decision that Mellanox or not.

They show a horrible software support and quality every time.

I’m very disappointed it.

But I’m also heard about industry 1st support for VRDMA in vSphere 6.5.

Keep wait to vSphere 6.5.

Ok I think I got all those corrections (might help someone later looking this up)

Driver VersionStorage ProtocolAdapter ModeAdapter FamilyVMware Ver Supported****Notes1.8.2.4IPoIB+iSCSI, SRPVPI Only,?ENCX-2,CX-3,CX-3 ProESXi5.x1.8.2.5IPoIB+iSCSI, SRPVPI Only,?ENCX-2,CX-3,CX-3 ProESXi6.01.8.3betaIPoIB+iSCSI, SRP, IPoIB+iSerVPI OnlyCX-2,CX-3,CX-3 ProESXi5.1(ESXi5.5 and 6.0 forced)1.9iSCSI, iSerEN OnlyCX-3,CX-3 ProESXi5.5,ESXi6.02IPoIB+iSCSIVPI,?ENCX-3,CX-3 ProESXi5.5,ESXi6.03iSCSIEN OnlyCX-3,CX-3 ProESXi5.5,ESXi6.0,ESXi6.54iSCSIEN OnlyCX-4ESXi5.5,ESXi6.0,ESXi6.5

Hi Jason,

Sorry for not answering sooner, here are answers to your questions, let me know if you have any further questions.

Jason wrote:

Also stated was that ESXi v6.5 includes drivers for CX-4 inbox (and CX-5 drivers are coming later this year) and that these drivers support ROCE and iSer when adapters are run in ethernet mode on an ethernet switch only.

[ES] RoCE yes. iSER is not part of Mellanox native driver. iSER is under evaluation with vmware for future ESXi releases either as an async driver or inbox native driver.

Jason wrote:

Because of the new native drivers included inbox for ESXi v6.5 the older VMKLinux drivers will not work any longer. Can we disable or remove these drivers and continue to use the older VMKLinux drivers similar to what is described here VMware Knowledge Base VMware Knowledge Base ?

[ES] vmware no longer supports vmklinux with ESXi 6.5, hence Mellanox have not verified the use of vmklinux drivers with ESXi 6.5.

Jason wrote:

Will there be any support for CX-3 and CX-3Pro adapters under the new Native device driver model for ESXi v6.5 and later? (and therefore support for iSer with CX-3 and CX-3Pro like you provide for CX-4 now (and for CX-5 in the future)?

[ES] ConnectX-3 and ConnectX-3 Pro have been added to the inbox native driver support with ESXi 6.5.

Jason wrote:

Will there be any support for IPoIB on the new Native device driver model or is this not being carried forward either?

[ES] InfiniBand (IB) is not supported by vmware. Previously, IB was supported with the use of Mellanox’s vmklinux driver. With the new native drivers we cannot add IB support, and it is up to vmware to add such support. That said, Mellanox will release in the upcoming June-July version support for InfiniBand over SR-IOV for ConnectX-4 and ConnectX-5 (note that management of ESXi is still via Ethernet, so the server will require both Ethernet and IB connectivity). With IB over SR-IOV you may use SRP, iSER or IPoIB in the guest OS.

Erez

Also finally in another side issue to this (but still relevant) what are people using on the Initiator side for SRP/iSer?

Now I am well aware you can run up a Linux distribution of your choice(again what are people using here) and install SCST or LIO on it but I am looking for a more ‘appliance like’ solution that has been tested. (as I need to get this all up and running ASAP) ESXi was always very picky about what iSCSI initiator it would work well with.

The best thing I have been able to currently find is ESOS - Enterprise Storage OS http://www.esos-project.com/ which is exactly the sort of thing I am looking for but I am interested in comments on how well it works or alternatives as I need to make the right choice first time and as quickly (and cheaply) as possible since its not a long term solution. (don’t want much do it )

I don’t need any management of the underlying disk storage (HDD and SSD) as I have hardware raid to handle that. (so as much as ZFS may be a good option I am not going there as it doesn’t suit the hardware I currently have from what I can work out)

I guess with the lack of SRP activity in the industry there are aren’t a huge number of choices.

Hi! mpogr,

Absolutely! Yes, it is.

Same procedure work well on ESXi 6.5, too…:)

There is more native driver concern RDMA on ESXi 6.5.

But my scripts also disable the all of native driver for RDMA on ESXi 6.5.

And you must uninstall inbox driver on ESXi 6.5, just reboot your ESXi 6.5 host after install Mellanox OFED 1.8.2.5.

But these all procudure not officiall, at your own risk!

Best Regards,

Jaehoon Choi

The steps you outlined (i believe) only work for connectx 3 or older cards. Correct me if I am wrong, but I am still searching for a solution for connectx4 cards. mlx4 vs mlx5 drivers…

For all of those who would like to watch the full presentation “INF 8469 iSCSI/iSER Hardware SAN Performance over the Converged Data Center” you can go to the url listed below:

VMworld_com Breakout Sessions - Playback Library https://www.vmworld.com/en/sessions/index.html https://www.vmworld.com/en/sessions/index.html

On this page under the splash is a link “View the Breakout Sessions” where you can register and get free access to view content from VMworld 2016. If you search on INF8469 you can locate the breakout session and watch the complete recording.

I have the same question as well. The drivers for connectx-4 have never really materialized for vmware and infiniband storage.

This is one of the reason I have given up on infiniband for our storage, dropping protocol support and dropping adapter support because newer versions appear. Just not worth it anymore.

Will we have the honour of someone from Mellanox responding to this one? They seem to be monitoring these forums pretty closely, so no response so far looks a bit strange…

I guess it is supported CX-4 VPI HCA with ETH mode & CX-4 ETH adapter.

Erez can you clarify what you mean by esxi 6.5 having roce support but not iser? I thought iser was just iscsi over rdma, and roce was rdma over converged ethernet. If we cannot use iser with roce what storage initiator driver is supported then with RoCE to connect back to a storage device?

KVM?

No! KVM also have many limitations.

For example EoIB, etc.

Infiniband communication rely on SM

  • Subnet Manager

SM consist of some components and API.

But SM architecture didn’t designed for hypervisor world.

Historically many problems were exist in vSphere environment.

1st. ESXi

When vSphere 4.x age VMware give us two choice.

ESX and ESXi

ESX consist hypervisor and OEMed Red Hat console.

ESXi consist hypervisor only.

Some IB tools didn’t work in ESXi host in my experience. But that IB tools work in ESX host nicely.

ESXi isn’t a general purpose kernel.

I think it cause a major IB driver porting problem.

2nd. Infiniband design itself!!!

Hypervisor control all communication guest vm and host network. RDMA have a kernel by-pass feature called zero copy or RDMA R/W.

This feature controlled by SM, but add hypervisor to this network, many complex modification must add to SM and IB APIs.

There isn’t IBTA standard yet.

This will be stadardize in near future.

3rd. RDMA storage protocol.

Infiniband specification concern all of RDMA and ULP protocols.

The improvement of Linux OFED is very fast.

No one know Which OFED version will be port to latest ESXi version.

Many complexity exist and must resolve issues in ESXi environment.

iSER also good candidate for ESXi RDMA protocol, but some critical problem exist.

I think that we must check the latest Linux OFED release note that include so many bugs, limitations.

Linux is a good platform but also suffer from IB’s unique limitations.

Conclusion.

I think IB is fastest and eiffient high speed protocol in the planet. But not ready to enterprise network environment yet.

Mellanox said in their product brouchur that they can support major OS environments.

But many case wasn’t.

Beta level driver, manual, bug, limitation, etc.

Absolutely!

Many time later all problem will be overcome with new standard and product.

But not now…

Two months later and still no reaction from Mellanox. They post lots of information here, mainly about new products, but what about taking care of existing products/customers?

I have seen the 2.4.0 drivers published under VPI/VMware driver section, but these are not new drivers, seems like a mistake corrected (these drivers previously appeared under ETH/VMware section). They do add official support for ESXi 6.0, but still no SRP/iSER. So, effectively, they allow using ConnectX-3/Pro VPI adapters as NICs over IB fabric (using IPoIB), but that’s it. There seems to be still no support at all(!) for ConnectX-IB and ConnextX-4 VPI adapters, which I find borderline insulting.

Any chance to get ANY reply from Mellanox staff?

What does it take to enable roce in esxi 6.5 with a connectx-4 card? I am guessing iscsi is the default vs iser and there may be additional config needed