VM Passthrough of MHQH19B-XTR

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I’m trying to passthrough a MHQH19B-XTR card to a Solaris 11.1 VM on VMware ESXi 5.1 but had no luck so far.

The core problem, based on my research, is that the card cannot handle the Function Level Reset (FLR) of the PCI-E bus when in passthrough mode.

I’ve tried flashing the 2.9.1200 firmware as well as adding the following line to “passthru.map” file on ESXi:

# Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s] 15b3 673c d3d0 false

The passthrough can be setup on the ESXi server but when the Solaris VM is booted it hangs. I’ve tried to passthrough it to a Windows 2012 VM and the VM shuts down immediately after boot.

I’m not sure if this is asked before but has anyone encountered this problem? Is there a solution to it?

2

I have same problem and drop the direct path-through.

I was tested vSphere 5.1, 5.1 U1 with OFED 1.8.1.0 for ESXi.

3

My test was completed now…

Look at the pictures below.

1361×959 181 KB

1325×943 166 KB

1366×987 180 KB

I’ll test more some tests and post again…

4

That’s a good news~!

Can you explain to me that your system configurations?

System Model - Vendor or not.

IB HCA mode and firmware version

Server Board Model

BIOS conguration include Intel vt-D, SR-IOV, etc

My all system based Super Micro X8DTN±FLR.

  • Not vendor product
5

I’m test with firmware 2.10.720 on CX-2 and vSphere 5.5.

Result was also failed…

Guest OS is Windows Server 2008 R2 SP1.

Mmmm…

That’s very difficult.

6

Heh, I guess you’re thinking of “VT” there:

x86 virtualization - Wikipedia, the free encyclopedia x86 virtualization - Wikipedia

“VT-D” isn’t as fundamental. It’s definitely needed for some use cases, but not everything.

Intel® Virtualization Technology for Directed I/O (VT-d): Enhancing Intel platforms for efficient virtualization of I/O … Data Center

7

I thought vt-D is essential for the whole virtualization platform? especially ESXi?

8

what version of the OFED driver did you use for the ESXi?

9

OK I tried flashing the 2.9.8350 firmware to no avail…

I also tried disabling “VT-d” but then ESXi does not allow me to passthrough any device at all so I have to keep it enabled.

Any suggestion?

10

I’ll try with fw.2.10.720.

But my WS2k12, 2k12R2 does go into endless reboot with BSOD. i’m also found some info about vt-D support on vSphere. I think WS2k8R2 and Red hat 6.4 support only.

I’ll try again this weekend.

So long…

11

Seems I’m using 1.8.0.0-1OEM.500.0.0.472560 according to “esxcli software vib list”.

@inbusiness what do you mean by “drop the direct path-through”?

12

There is a some problem on vSphere ESXi support Intel vt-D.

If you add a “pciPassthru0.maxMSIXvectors = 31” option in *.vmx configuration then

you’ll boot Windows VM with passthrough IB-HCA successfully.

But you’ll find a problem in windows device manager with yellow mark.

It’s a need some HCA firmware update and vSphere ESXi update, too…

VMware KB: Virtual machines with Windows 2008 and later guest OS fail to power on if Intel Ethernet Server Adapter X… VMware Knowledge Base

13

It’s “LSI Logic Parallel”. It is strange as I thought it should support CX-2 out of box but turns out not…

Now the link is successfully established but I see there’s a large latency, i.e. 600-2000ms using ping test…whereas other type of VMs only show <1ms ping. Definitely something wrong. Maybe the support in the new version is not complete?

14

Probably you’re using an old Firmware.

Update your HCA Firmware image to the latest one and try again, this should solve the FLR issue.

15

2.9.1200 is released on 07-May-12 whereas 2.9.8350 is released on 07-Dec-11?

In addition 2.9.1200 release notes mentions 2 bugs fixed:

"

PCIe FunctionLevelReset occasionally failed when the driver was down

VMware driver failed to load when attached to Virtual Function one

"

whereas 2.9.8350 release notes mantions nothing related to FLR/virtualization.

So that’s why I didn’t try the 2.9.8350 FW, although the number seems higher…maybe I missed some information somewhere?

16

FLR=Function Level Reset

Some reference I found:

'Re: Help with strange problem passing mlx4 device into kvm guests' - MARC 'Re: Help with strange problem passing mlx4 device into kvm guests' - MARC

and

Re: mellanox mlx4_core and SR-IOV Re: mellanox mlx4_core and SR-IOV

I’m already at FW 2.9.1200 which is the latest and said to fix this problem but actually not.

17

Hi xlx01,

Can you try the (intermediate) FW 2.9.8350 from:

OEM Firmware Downloads OEM Firmware Downloads

and let us know if it solves the problem.

18

Nice that makes it boot! I’m using WS2012 tho. I’ll try on my Solaris VM soon and report back!

19

That mean I decide don’t use Intel vt-D anymore…

I’m wait the iSER or vRDMA…

20

What’s the FLR issue?

Do you mean my system board SMC X8DTN±FLR?