MCX654105A-HCAT on Linux shows an ib0 and ib1 and appears as two HCA's, mlx5_0 and mlx5_1, which do I use?

[dcunited]

Hi all, we inherited an HPC system where each node contains a single physical port, with two x16 cards for socket direct (this RHEL 8). lspci shows both the main board and the auxillary card as it should, but we’re confused because we can’t find any documentation on best practices for the two ib0 and ib1 interfaces that come up. We’ve been just using ib0, and leaving ib1 with no ip address assigned.

lspci shows what the driver installation instructions say we should see, but nowhere can i find what if any preferred interface we should use, or if there’s a differernce. Should they be bonded? Is the auxillary card only there for supplemental power? My understanding was that it was so a dual socket machine could connect through different ones, but would they need their own ip addresses?

7f:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
ac:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]

CA ‘mlx5_0’
CA type: MT4123
Number of ports: 1
Firmware version: 20.35.3006
Hardware version: 0
Node GUID: 0x0c42a1030099939c
System image GUID: 0x0c42a1030099939c
Port 1:
State: Active
Physical state: LinkUp
Rate: 200
Base lid: 25
LMC: 0
SM lid: 5
Capability mask: 0xa651e848
Port GUID: 0x0c42a1030099939c
Link layer: InfiniBand
CA ‘mlx5_1’
CA type: MT4123
Number of ports: 1
Firmware version: 20.35.3006
Hardware version: 0
Node GUID: 0x0c42a1030099939e
System image GUID: 0x0c42a1030099939c
Port 1:
State: Active
Physical state: LinkUp
Rate: 200
Base lid: 36
LMC: 0
SM lid: 5
Capability mask: 0xa641e848
Port GUID: 0x0c42a1030099939e
Link layer: InfiniBand

ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 192.168.30.11 netmask 255.255.255.0 broadcast 192.168.30.255
inet6 fe80::e42:a103:99:939c prefixlen 64 scopeid 0x20
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
infiniband 00:00:00:4C:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
RX packets 1293992 bytes 1493855506 (1.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2930310 bytes 3604531200 (3.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ib1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
infiniband 00:00:01:F0:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
RX packets 38052 bytes 3805232 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[dwaxman]

The aux card exposes another interface which can be used separately or in a static bonding configuration (Recommend using balanced-rr bonding mode).

For supporting the SD card in Infiniband – OpenSM needs to be configured with virtualization support (it should be default but in any case ensure the following):
Edit the opensm.conf file (MLNX OFED default location - /etc/opensm/opensm.conf) - Change the value for virt_enabled to 2, save the file and restart the SM.

A few other things to note:

1. ‘ibdev2netdev -v’ command will exhibit the primary interface and the aux interface.

2. to validate performance need to run multi device RDMA test that’ll utilize both interfaces (you will need to sum the results presented for each device in the test below, to see the

Server side :
run_perftest_multi_devices -d mlx5_0,mlx5_1 -c 0,1 --cmd “ib_write_bw”

Client side:
run_perftest_multi_devices -d mlx5_0,mlx5_1 -c 0,1 --cmd “ib_write_bw --report_gbits” --remote

Remember to set the PCIe MaxReadReq to 4K
1. Check current max read value:

lspci -s 04:00.0 -vvv | grep MaxReadReq

2. Query the value

setpci -s 04:00.0 68.w

2020
3. Change the first digit of the output in the step above to an acceptable value. In our case, its 5. So, 2020 will be 5020.

setpci -s 04:00.0 68.w=5020

The acceptable values are:
0 - 128B, 1 - 256B, 2 - 512B, 3 - 1024B, 4 - 2048B and 5 - 4096B.

3. Addional settings – iommu in grub (and/or in BIOS – different systems behave differently)
   Set “iommu=pt” on the kernel grub menu.
   For example:
   BOOT_IMAGE=/boot/vmlinuz-4.13.0-16-generic.efi.signed root=UUID=91e6c1c7-271e-405b-a498-ff799fd0fa05 ro crashkernel=auto rhgb console=tty0 console=ttyS0,115200 iommu=pt