Recently, we are investigating a possibility of running the IPoIB/RDS protocol on the RoCE interface of IBM LinuxONE Server (which can running mainstream Enterprise Linux system, such as RHEL, SLES and UBUNTU).
It integrated a Mellanox MT27500 ConnectX-3 card as a 10GbE RoCE interface.
But i had found that the inbox driver with the redhat linux distribution seems to be incomplete, the most apparent evidence to support this are the “InfiniBand Support” and “infiniband-diags” subgroup are missing in the repository.
So, i turn to this forum for help.
Does anyone know the process to build the driver on a s390x arch system?
Here is the reported issues in the compat-rdma.rpmbuild.log
.....................
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/vport.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/sriov.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.o
/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.c: In function 'mlx4_en_poll_rx_cq':
/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.c:1191:3: error: implicit declaration of function 'irq_desc_get_irq_data' [-Werror=implicit-function-declaration]
aff = irq_desc_get_irq_data(cq->irq_desc)->affinity;
^
/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.c:1191:44: error: invalid type argument of '->' (have 'int')
aff = irq_desc_get_irq_data(cq->irq_desc)->affinity;
^
cc1: some warnings being treated as errors
make[3]: *** [/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4/en_rx.o] Error 1
make[2]: *** [/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx4] Error 2
make[2]: *** Waiting for unfinished jobs....
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/fs_core.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/rl.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/wq.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_main.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_common.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_fs.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_tx.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_rx.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_clock.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/vxlan.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_tc.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.o
CC [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/en_dcbnl.o
LD [M] /var/tmp/OFED_topdir/BUILD/compat-rdma-4.8/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.o
make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/compat-rdma-4.8] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.10.0-514.10.2.el7.s390x'
make: *** [kernel] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.UO3eLF (%build)
RPM build errors:
user vlad does not exist - using root
user vlad does not exist - using root
Bad exit status from /var/tmp/rpm-tmp.UO3eLF (%build)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Regards
FuLong
But those drivers are for Linux on IBM Power Platform (known as the powerlinux, arch is ppc64 or ppc64le) and Linux on x86 platform, not for Linux on IBM z Systems (known as zlinux, arch is s390x) ,
I think the system architecture is totally different.
I checked it internally , since IBM z system is developed and maintained by IBM they should provide drivers that are compatible to work with our cards.
The MLNX-OFED driver is distributed per your operating system and HCA type, what is the OS you are currently using? can be checked by invoking : cat /etc/redhat-release
For your refernce please also see our driver support matrix:
Look through the driver support matrix, no match found yet.
The card integrated on the IBM z13 machine is “Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]”.
Two cards equipped, two ports on each card, only one port is cabled for now.
I had set up two test env (one is RHEL 7.2, another is RHEL 7.3), actually, we can run RHEL 7.4 (the version just released by Redhat) and SLES & the latest UBUNTU for s390x arch also.
I also try to build the modules and rpms on another similar env (with optional repositories) with the latest OFED-4.8, but no luck till now, don’t know if the code need to be adjusted or not?
