It looks like you are trying to install OFED 1.5.3, not 1.5.4
If you are using connectx2/3 cards you can simply install OFED 2.0
If you are using the older 10Gb cards, you’ll need to install 1.5.4. (1.5.3 may work, but I have not tried to get this working). There are some conflicts you will need to resolve first in the ofa_kernel tree.
Remove line 9 (definition of ‘IS_ERR_OR_NULL’) from:
I don’t believe we have a product/solution that would be similar to this other than SR-IOV which going forward I believe will be the preferred implementation used for this effect.
Is SR-IOV for DomU (guest) supported (with MLNX_OFED 2.0) in XenServer for all ConnectX cards?
I am not sure it will be all ConnectX cards. ConnectX is EOL but I am confident it supports ConnectX2 and ConnectX3 generations.
We have been using Infiniband with Xenserver (XS) since v6.0 below are our findings:
OFED1.5.4 compiles and runs on XS v6.0. IPoIB is the only functioning transport mechanism. SDP, SRP and iSER DO NOT WORK. They are naming conflicts between the OFED stack and Xen kernel which prevent the SRP modules from loading, and whilst the iSER module loads, when you attempt to create a connection using iSER as transport it crashes.
This is due to a bug in the kernel tree that was introduced in v6 and not rectified (as infiniband is outside vendor support for Xenserver), I note that iSER reportedly was working correctly in XS 5.6.
In XS 6.1 OFED fails to compile due to duplication of some naming issues in the network stack, we managed to hack 1.5.4 to compile. I note this is when Mellanox introduced OFED 2.0, however this only supports the later model cards, so if you run 10Gb infiniband product, this won’t work. Once again IPoIB is the only transport that works.
In XS 6.2, our hacked version of OFED 1.5.4 compiles fine, however OFED 2.0 breaks, I believe there is a method within the 2.0 packaging to compile against a new kernel, however we did not try this.
We have been mulling running dom0 passthrough and setting up storage on a virtual machine, however this is currently a hack and is not past alpha level yet.
In summary, It would be really nice for Mellanox to work with the new (and improved) Xen opensource project to get infiniband drivers into the kernel. It makes sense for the best storage transport to function in the most deployed Hypervisor.
following your word, I have tried to compile OFED v 1.5.4 into xenserver environment, but:
./mlnx_add_kernel_support.sh stop with error: “kernel-ib was not created”, but kernel_ib and kernel_ib_devel rpm are created and I found these into subdir /root-of-mlx-tmp-work/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/centos-release-5-7.el5.centos/i686/
At the end of my test, I have get compiled package “kernel-ib” and “ofed-script” and I have installed these into xenserver.
During boot somes modules aren’t loaded, error displayed on openibd startup are:
When you compile OFED 1.5.4 on the DDK it will build out all the required .rpm’s for you to install on your production systems.
To be honest, if you can turn back now, i would not use Infiniband with XenServer. Performance with 10Gb Infiniband and IPoIB will be equivalent to 4Gb Fibre Channel. If you use Connectx2 cards and switches performance is similar to 10Gb Ethernet (IPoIB).
If you use a hypervisor with native support with infiniband (KVM, HyperV), you’ll be able to use your infrastructure to its potential.
If you can wait, Xenserver will be released as a toolstack sometime this year. This means it will work with latest vanilla kernel which has native support for infiniband built in.