I remember when x86 virtualization solution - VMware vSphere, Microsoft Hyper-V, Citrix Xen, Linux KVM, etc - appear on market, all works good.
Many administrator can increase server utilization that decrease host quantity and save money to operation price.
But by increasing VM density, performance problem was appeared on hypervisor host.
x86 Hypervisor layer is located between hardware and Guest OS.
All of storage, network I/O passed via hypervisor.
Therefore almost Virtualization admin worry about it that cause performance problem.
Some vendor start launch solution for it like SR-IOV or etc.
But I’m disappointed, because there are so many limitation in their solutions.
I remember that I saw a RDMA protocol concept at for the first time from 7~8 yeares ago.
I think the RDMA protocol resolve this problem on hypervisor.
They can support all of virtualization solution’s features like vMotion, etc.
When vSphere 4.x era all works great with Mellanox VPI solutions.
One Mellanox VPI HCA act as vHBA, vNIC like FCoE CNA adapter.
That feature reduce cable count and configuration complexity also reduce operation budget.
When I change the infrastructure to ConnectX-1, 2 QDR configuration, that show me a extreme performance, too.
But when VMware vSphere 5.0 launched all hope was gone away ~
VMware release a new native driver architecture on last ESXi 5.x.
VMware says native driver will be a arhitecture that show us High performance and Efficiency.
But there is important question that Is VMware’s native driver architecture show us performance and efficiency like RDMA?
Absolutely!
Increase I/O efficiency on hypervisor will be a good concept.
But I don’t think VMware native driver architecture show us same performance and efficiency like RDMA.
RDMA is kernel bypass architecture that bypass also hypervisor.
It’s a difficult protocol to configure then traditional ethernet, fibre channel architecture, but it most valuable archietecture.
Let’s go to think about current VMware’s native driver architecture in real world.
VMware’s native driver architecture show us 2 of major problems currently like below
01. Performance Problem
Some people on the planet said VMware’s native driver show a problems like native AHCI driver that show a extreme low performance.
Disable inbox AHCI native driver then reboot each ESXi 6.x, ESXi 6.x use previous vKernel AHCI driver that resolve performance problem.
- I think that VMware will release patch for it later.
I’m not a AHCI user…:)
02. Remove useful functions
Intel X520 native driver can’t support SR-IOV and FCoE like previous vKernel driver for Intel X520.
There is a same solution that disable inbox native X520 driver, reboot ESXi 6.x host then ESXi 6.x host use previous vKernel driver.
Mellanox VPI HCA also affected, too.
All of features based on vSphere 4.0 was gone away!
A. Multiple IPoIB port similar SR-IOV
In fact multiple IPoIB port works different way…:)
Because VLAN configuration works greatly with IPoIB, not a problems but you must configure IB partition with Hex code…:(
B. RDMA vHBA
Yes! You know SRP initiator!
I’m also use Mellanox 1.8.3 IPoIB iSER concept driver.
That show me a extreme performance like SRP initiator.
But this is concept driver that caused VMware Host Profile creation error!
Problem B is major problem on Mellanox VPI user or admin.
Because lack of RDMA vHBA - SRP or iSER on Infiniband - driver support on ESXi 6.x, many admin must make a dicision what is the best protocol for their storage now.
I’m also saw a PSOD problem randomly with Mellanox OFED 1.8.2.x SRP initiator with ZFS COMSTAR SRP target on ESXi 6.x.
That cause I’m also change infrastructure to 10Gb FCoE temporarily almost 1 year.
Here is a my question.
Do you think VMware’s native driver archietecture show us a performance and efficiency like RDMA?
P.S
I’m completed test with Mellanox OFED 1.8.2.x on ESXi 6.0, 6.5 weekend.
I saw a VMware native driver problems and resolutions in web sites.
Double check the inbox Mellanox driver and native driver list on ESXi 6.0, 6.5.
I was success operate SRP initiator on ESXi 6.0, 6.5 host with OmniOS ZFS COMSTAR SRP target.
Some extreme storage I/O load test was completed successfully with VMware I/O Analyzer like below
3 ESXi 6.x SRP initiator to different LUN - each worker VM reside on different LUNs - on same OmniOS ZFS COMSTAR SRP Target
3 ESXi 6.x SRP initiator to same LUN - 3 worker VMs reside on same LUN - on same OmniOS ZFS COMSTAR SRP Target
Every VM works great like Mellanox OFED 1.8.2.x on ESXi 5.x.
Some times later I’ll write up how can I build a SRP initiator on ESXi 6.x
P.S 2
I prefer Infiniband iSER then SRP.
SRP is very ancient protocol that don’t have a disconnection from SRP target function.
That cause when reboot our ESXi host log a symbol error on my SX6036G port status and delay reconnect to SRP target
that can’t support auto VM start properly.
P.S 3
RoCE configuration guide on ESXi 6.x is also very unclear.
Sometimes I saw that they write a Linux configuration scripts on ESXi guide…:(