I’m trying to passthrough a MHQH19B-XTR card to a Solaris 11.1 VM on VMware ESXi 5.1 but had no luck so far.
The core problem, based on my research, is that the card cannot handle the Function Level Reset (FLR) of the PCI-E bus when in passthrough mode.
I’ve tried flashing the 2.9.1200 firmware as well as adding the following line to “passthru.map” file on ESXi:
# Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s] 15b3 673c d3d0 false
The passthrough can be setup on the ESXi server but when the Solaris VM is booted it hangs. I’ve tried to passthrough it to a Windows 2012 VM and the VM shuts down immediately after boot.
I’m not sure if this is asked before but has anyone encountered this problem? Is there a solution to it?
I have same problem and drop the direct path-through.
I was tested vSphere 5.1, 5.1 U1 with OFED 220.127.116.11 for ESXi.
My test was completed now…
Look at the pictures below.
I’ll test more some tests and post again…
That’s a good news~!
Can you explain to me that your system configurations?
System Model - Vendor or not.
IB HCA mode and firmware version
Server Board Model
BIOS conguration include Intel vt-D, SR-IOV, etc
My all system based Super Micro X8DTN±FLR.
I’m test with firmware 2.10.720 on CX-2 and vSphere 5.5.
Result was also failed…
Guest OS is Windows Server 2008 R2 SP1.
That’s very difficult.
Heh, I guess you’re thinking of “VT” there:
x86 virtualization - Wikipedia, the free encyclopedia http://en.wikipedia.org/wiki/X86_virtualization
“VT-D” isn’t as fundamental. It’s definitely needed for some use cases, but not everything.
Intel® Virtualization Technology for Directed I/O (VT-d): Enhancing Intel platforms for efficient virtualization of I/O … http://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices
I thought vt-D is essential for the whole virtualization platform? especially ESXi?
what version of the OFED driver did you use for the ESXi?
OK I tried flashing the 2.9.8350 firmware to no avail…
I also tried disabling “VT-d” but then ESXi does not allow me to passthrough any device at all so I have to keep it enabled.
I’ll try with fw.2.10.720.
But my WS2k12, 2k12R2 does go into endless reboot with BSOD. i’m also found some info about vt-D support on vSphere. I think WS2k8R2 and Red hat 6.4 support only.
I’ll try again this weekend.
Seems I’m using 18.104.22.168-1OEM.500.0.0.472560 according to “esxcli software vib list”.
@inbusiness what do you mean by “drop the direct path-through”?
There is a some problem on vSphere ESXi support Intel vt-D.
If you add a “pciPassthru0.maxMSIXvectors = 31” option in *.vmx configuration then
you’ll boot Windows VM with passthrough IB-HCA successfully.
But you’ll find a problem in windows device manager with yellow mark.
It’s a need some HCA firmware update and vSphere ESXi update, too…
VMware KB: Virtual machines with Windows 2008 and later guest OS fail to power on if Intel Ethernet Server Adapter X… http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2032981
It’s “LSI Logic Parallel”. It is strange as I thought it should support CX-2 out of box but turns out not…
Now the link is successfully established but I see there’s a large latency, i.e. 600-2000ms using ping test…whereas other type of VMs only show <1ms ping. Definitely something wrong. Maybe the support in the new version is not complete?
Probably you’re using an old Firmware.
Update your HCA Firmware image to the latest one and try again, this should solve the FLR issue.
2.9.1200 is released on 07-May-12 whereas 2.9.8350 is released on 07-Dec-11?
In addition 2.9.1200 release notes mentions 2 bugs fixed:
PCIe FunctionLevelReset occasionally failed when the driver was down
VMware driver failed to load when attached to Virtual Function one
whereas 2.9.8350 release notes mantions nothing related to FLR/virtualization.
So that’s why I didn’t try the 2.9.8350 FW, although the number seems higher…maybe I missed some information somewhere?
Can you try the (intermediate) FW 2.9.8350 from:
and let us know if it solves the problem.
Nice that makes it boot! I’m using WS2012 tho. I’ll try on my Solaris VM soon and report back!
That mean I decide don’t use Intel vt-D anymore…
I’m wait the iSER or vRDMA…
What’s the FLR issue?
Do you mean my system board SMC X8DTN±FLR?