I am trying to setup CX5 RDMA in between two KVM guests (one in each physical node) but failed with rping segfault ... in librdmacm.so.1.1.20.2. Is there any articles here I can follow to resolve the issue?

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
, , , ,
1

The systems I used are with Intel Xeon and the two servers have identical configuration. Both of them have CX5 in the same PCI slot and the servers and KVM guests OS are OEL7.6 with UEK5 kernel. I could do rping in between the two bare metal servers successfully. Then I enabled SRIOV of CX5 and created two VFs on the same CX5 port. Then assigned one CX5 VF to the VM each. Started VMs and setup CX5 in VMs. They can ping each other fine but rping in initiator side failed with segfault:

#target

[root@scaoda8m020c2n1 ~]# rping -s -a 192.168.105.88 -p 9999

#initiator

[root@scaoda8m021c2n1 ~]# rping -c -Vv -C5 -a 192.168.105.88 -p 9999

Segmentation fault (core dumped)

[root@scaoda8m021c2n1 ~]# dmesg |tail -5

[ 226.688321] nr_pdflush_threads exported in /proc is scheduled for removal

[ 229.665968] capability: warning: `turbostat.4.14.’ uses 32-bit capabilities (legacy support in use)

[ 416.992990] rping[4305]: segfault at 0 ip 00007f53033f2c10 sp 00007f53003bbb90 error 4 in librdmacm.so.1.1.20.2[7f53033eb000+15000]

[ 580.196483] udaddy[5958]: segfault at 0 ip 00007faa0a9acc10 sp 00007ffcae88edc0 error 4 in librdmacm.so.1.1.20.2[7faa0a9a5000+15000]

[ 2276.257078] rping[8893]: segfault at 0 ip 00007f79ad279c10 sp 00007f79aa242b90 error 4 in librdmacm.so.1.1.20.2[7f79ad272000+15000]

Please help and thanks. -Ken

2

Hi Chen,

No I am not using MLNX_OFED driver. I am using drivers and utilities from Oracle Linux 7 RDMA for UEK5 yum site. We don’t have MLNX_OFED for Linux 7 UEK5. The MLNX_OFED driver is supposed to be in Linux 7 UEK5 as I understand. I am using the same drivers and utilities in both hosts and VMs but the rping failed in VMs. I have gone through enabling SR-IOV and creating VFs and PCI passthrough VF to VM. I could see the CX5 with lspci in the KVM guests such as:

[root@scaoda8m020c2n1 yum.repos.d]# lspci |grep Mella

00:08.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]

But got Segfault when running rping as initiator:

[root@scaoda8m020c2n1 yum.repos.d]# rping -c -Vv -C5 -a 192.168.105.88 -p 9999

Segmentation fault (core dumped)

[root@scaoda8m020c2n1 yum.repos.d]# dmesg |tail -5

[ 24.452859] 8021q: adding VLAN 0 to HW filter on device eth0

[ 24.453669] 8021q: adding VLAN 0 to HW filter on device ens8

[ 6664.707026] rping[2830]: segfault at 0 ip 00007f5aec234c10 sp 00007f5ae91fdb90 error 4 in librdmacm.so.1.1.20.2[7f5aec22d000+15000]

[ 6668.740105] nr_pdflush_threads exported in /proc is scheduled for removal

[ 6671.878428] capability: warning: `turbostat.4.14.’ uses 32-bit capabilities (legacy support in use)

Wonder did I miss installed anything in the RDMA software stack in the KVM guests?

Ken

3

Hi Ken,

Are you working with MLNX_OFED driver?

Please review the User Manual - Single Root IO Virtualization (SR-IOV)

https://docs.mellanox.com/pages/viewpage.action?pageId=34265067

Make sure to follow all the required steps to configure SRIOV.

Regards,

Chen

4

I suspect that I hit the known issue: 1047616

Description: When node GUID of a device is set to zero (0000:0000:0000:0000), RDMA_CM user space application may crash.

I tried to follow the Workaround: Set node GUID to a nonzero value. But found I could not mstflint set node_guid of VF:

[root@oak ~]# lspci -D |grep Mell

0000:af:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

0000:af:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

0000:af:00.2 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]

0000:af:00.3 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]

[root@oak ~]# mstflint -d af:00.2 -guid 0002c9000abcdef0 sg

-E- Cannot open Device: af:00.2. Operation not supported. MFE_CR_ERROR

[root@oak ~]# mstflint -d 0000:af:00.2 -guid 0002c9000abcdef0 sg

-E- Cannot open Device: 0000:af:00.2. Operation not supported. MFE_CR_ERROR

[root@oak ~]# ibv_devices

device node GUID

mlx5_2 0000000000000000

mlx5_0 506b4b0300d71358

mlx5_1 0000000000000000

I tried set them in PF, it could be done but still only the PF has non-zero node_guid.

5

Verified that after updated firmware to version 16.28.2006 and used ip link set mac address of the VF in host. Although the VF node_guid showed in host ibv_devices were still 0 but showed non-zero value in the KVM guest. And then rping running successfully. Confirmed that I hit the known issue 1047616.