Where i can get or how to build the driver for Linux on IBM z system?

Hi Gurus,

I’m really newbie to this forum!

Recently, we are investigating a possibility of running the IPoIB/RDS protocol on the RoCE interface of IBM LinuxONE Server (which can running mainstream Enterprise Linux system, such as RHEL, SLES and UBUNTU).

It integrated a Mellanox MT27500 ConnectX-3 card as a 10GbE RoCE interface.

But i had found that the inbox driver with the redhat linux distribution seems to be incomplete, the most apparent evidence to support this are the “InfiniBand Support” and “infiniband-diags” subgroup are missing in the repository.

So, i turn to this forum for help.

Does anyone know the process to build the driver on a s390x arch system?

Any help are appreciate!

Regards

FuLong

Here is the reported issues in the compat-rdma.rpmbuild.log


.....................

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/vport.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/sriov.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.o

/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.c: In function 'mlx4_en_poll_rx_cq':

/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.c:1191:3: error: implicit declaration of function 'irq_desc_get_irq_data' [-Werror=implicit-function-declaration]

aff = irq_desc_get_irq_data(cq->irq_desc)->affinity;

^

/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.c:1191:44: error: invalid type argument of '->' (have 'int')

aff = irq_desc_get_irq_data(cq->irq_desc)->affinity;

^

cc1: some warnings being treated as errors

make[3]: *** [/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.o] Error 1

make[2]: *** [/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4] Error 2

make[2]: *** Waiting for unfinished jobs....

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/fs_core.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/rl.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/wq.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_main.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_common.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_fs.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_tx.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_rx.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_clock.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/vxlan.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_tc.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.o

CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_dcbnl.o

LD [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.o

make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8] Error 2

make[1]: Leaving directory `/usr/src/kernels/3.10.0-514.10.2.el7.s390x'

make: *** [kernel] Error 2

error: Bad exit status from /var/tmp/rpm-tmp.UO3eLF (%build)

RPM build errors:

user vlad does not exist - using root

user vlad does not exist - using root

Bad exit status from /var/tmp/rpm-tmp.UO3eLF (%build)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Regards

FuLong

Hi Wang,

Please try the following IBM’s best recipe supported solution that was tested with our drivers:

IBM Network Adapter Cards & Switch Systems Software and Firmware IBM Network Adapter Cards & Switch Systems Software and Firmware

Let me know if this is helpful.

Thanks,

Karen.

Karen,

I would try it later for a hope!

But those drivers are for Linux on IBM Power Platform (known as the powerlinux, arch is ppc64 or ppc64le) and Linux on x86 platform, not for Linux on IBM z Systems (known as zlinux, arch is s390x) ,

I think the system architecture is totally different.

Regards

FuLong

HI Wang,

I checked it internally , since IBM z system is developed and maintained by IBM they should provide drivers that are compatible to work with our cards.

Thank you,

Karen.

Hi Fulong,

The MLNX-OFED driver is distributed per your operating system and HCA type, what is the OS you are currently using? can be checked by invoking : cat /etc/redhat-release

For your refernce please also see our driver support matrix:

InfiniBand OS Distributors InfiniBand OS Distributors

Thanks ,

Karen.

Karen,

Thanks for the infor!

Look through the driver support matrix, no match found yet.

The card integrated on the IBM z13 machine is “Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]”.

Two cards equipped, two ports on each card, only one port is cabled for now.

I had set up two test env (one is RHEL 7.2, another is RHEL 7.3), actually, we can run RHEL 7.4 (the version just released by Redhat) and SLES & the latest UBUNTU for s390x arch also.

I also try to build the modules and rpms on another similar env (with optional repositories) with the latest OFED-4.8, but no luck till now, don’t know if the code need to be adjusted or not?

Regards

FuLong

Hi Wang,

Our driver distribution supports x86_64/PPC64 (Power8)/PPC64LE (Power8)/ARMv8 architectures.

If there is specific platform related to IBM you may need to approach IBM support for assistance.

Thank you,

Karen.

Thanks Karen,

So, No guys from IBM z Systems development ever reach out to your company for the driver development for Linux on IBM z System?

Regards

FuLong

Karen,

Forgot one thing, i think there are official technical supports on the platform of x86_64/PPC64 (Power8)/PPC64LE (Power8)/ARMv8 architect, right?

Regards

Fulong