Hi, I’m using a ConnectX3-VPI dual-port card in Ethernet mode. It’s setup as SRIOV device and all VMs living on the same port but in different VFs can talk to each other. However, communication between VFs on different ports of the same card doesn’t work. Is this by design or am I missing something? In case this feature isn’t supported on X3 cards, what about newer generations? Thanks!
Hi Fabian,
Just to clarify, are you saying that VM’s that have VF’s belonging to different PF (ports) cannot ping each others?
Are all VF’s regarding which ports they belonging to on the same subnet? If I am correct, are you able to ping if the VF’s belonging to different ports are part of a different subnet?
Sophie.
Hi Sophie,
that’s correct, VF’s belonging to different PF cannot ping each other, even when being in the same subnet and VLAN.
However, it’s even more strange. I can’t seem to see VFs (i.e. lspci) of the second port in VMs at all. Maybe some of my settings are entirely broken?
Here is some data regarding my config (24VFs, configured as 8/8/8):
Device #1:
Device type: ConnectX3
Device: /dev/mst/mt4099_pciconf0
Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 24
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 None(0)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 False(0)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 None(0)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)
2c:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
2c:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:00.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:00.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:00.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:00.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:00.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:00.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
2c:01.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:01.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:02.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
2c:03.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (rev ff)
1: enp44s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:25:8B:10:72:c0 brd ff:ff:ff:ff:ff:ff
vf 0 MAC f2:78:44:d1:c1:2a, vlan 4095, spoof checking on, link-state enable
vf 1 MAC 00:25:8B:10:72:d1, vlan 4095, spoof checking on, link-state enable
vf 2 MAC 00:25:8B:10:72:d2, vlan 4095, spoof checking on, link-state enable
vf 3 MAC 00:25:8B:10:72:d3, vlan 4095, spoof checking on, link-state enable
vf 4 MAC 00:25:8B:10:72:d4, vlan 100, spoof checking on, link-state enable
vf 5 MAC 00:25:8B:10:72:d5, vlan 100, spoof checking on, link-state enable
vf 6 MAC 00:25:8B:10:72:d6, vlan 100, spoof checking on, link-state enable
vf 7 MAC 00:25:8B:10:72:d7, vlan 100, spoof checking on, link-state enable
vf 8 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 9 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 10 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 11 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 12 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 13 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 14 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 15 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 16 MAC 00:25:8B:10:72:f0, vlan 100, spoof checking on, link-state enable
vf 17 MAC 00:25:8B:10:72:f1, vlan 100, spoof checking on, link-state enable
vf 18 MAC 00:25:8B:10:72:f2, vlan 100, spoof checking on, link-state enable
vf 19 MAC 00:25:8B:10:72:f3, vlan 100, spoof checking on, link-state enable
vf 20 MAC 00:25:8B:10:72:f4, vlan 100, spoof checking on, link-state enable
vf 21 MAC 00:25:8B:10:72:f5, vlan 100, spoof checking on, link-state enable
vf 22 MAC 00:25:8B:10:72:f6, vlan 100, spoof checking on, link-state enable
vf 23 MAC 00:25:8B:10:72:f7, vlan 100, spoof checking on, link-state enable
2: enp44s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:25:8B:10:72:c1 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 5 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 6 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 7 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 8 MAC 00:25:8B:10:72:e0, vlan 100, spoof checking on, link-state enable
vf 9 MAC 00:25:8B:10:72:e1, vlan 100, spoof checking on, link-state enable
vf 10 MAC 00:25:8B:10:72:e2, vlan 100, spoof checking on, link-state enable
vf 11 MAC 00:25:8B:10:72:e3, vlan 100, spoof checking on, link-state enable
vf 12 MAC 00:25:8B:10:72:e4, vlan 100, spoof checking on, link-state enable
vf 13 MAC 00:25:8B:10:72:e5, vlan 100, spoof checking on, link-state enable
vf 14 MAC 00:25:8B:10:72:e6, vlan 100, spoof checking on, link-state enable
vf 15 MAC 00:25:8B:10:72:e7, vlan 100, spoof checking on, link-state enable
vf 16 MAC 00:25:8B:10:72:f8, vlan 100, spoof checking on, link-state enable
vf 17 MAC 00:25:8B:10:72:f9, vlan 100, spoof checking on, link-state enable
vf 18 MAC 00:25:8B:10:72:fa, vlan 100, spoof checking on, link-state enable
vf 19 MAC 00:25:8B:10:72:fb, vlan 100, spoof checking on, link-state enable
vf 20 MAC 00:25:8B:10:72:fc, vlan 100, spoof checking on, link-state enable
vf 21 MAC 00:25:8B:10:72:fd, vlan 100, spoof checking on, link-state enable
vf 22 MAC 00:25:8B:10:72:fe, vlan 100, spoof checking on, link-state enable
vf 23 MAC 00:25:8B:10:72:ff, vlan 100, spoof checking on, link-state enable
Hi Fabian,
- Please provide the outputs for: #flint -d q
- #[ofed_info -s]
- What is the OS version on the Hypervisor and VM?
- Can you please let me know if you have followed the below documents to configure SR-IOV:
https://community.mellanox.com/s/article/howto-configure-sr-iov-vfs-on-different-connectx-3-ports
https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-3-with-kvm–ethernet-x
Thanks,
Namrata.
Hi Namrata,
1: #flint -d /dev/mst/mt4099_pciconf0 q
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 0002c90300XXXXXX 0002c90300XXXXXX 0002c90300XXXXXX 0002c90300XXXXXX
MACs: 0002c9XXXXXX 0002c9XXXXXX
VSD:
PSID: MT_1090120019
2: I don’t have OFED installed.
3: The host OS is Proxmox 6.0-11 with Kernel: Linux 5.0.21-3-pve #1 SMP PVE 5.0.21-7 (Mon, 30 Sep 2019 09:11:02 +0200) x86_64 GNU/Linux
4: I have used those documents with the exception of OFED. Instead of setting up the number of VFs in /etc/modprobe.d/mlx4.conf, they had to be configured in /etc/modprobe.d/mlx4_core.conf to show up.
As you can see from previously posted output, the NIC operates in Ethernet mode and is configured to provide 24VFs (8/8/8).
From my testing it seems, only the first 8VFs ever actually work (indicated by the “(rev ff)” as given in my first reply).
DMESG also shows:
[ 10.206495] mlx4_core: Initializing 0000:2c:03.0
[ 10.209604] mlx4_core 0000:2c:03.0: Skipping virtual function:24
[ 10.212738] mlx4_core 0000:2c:00.0: Running in master mode
[ 10.215878] mlx4_core 0000:2c:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 10.357005] mlx4_core: Initializing 0000:2c:00.1
[ 10.360169] mlx4_core 0000:2c:00.1: enabling device (0000 → 0002)
[ 10.364149] mlx4_core 0000:2c:00.1: Skipping virtual function:1
[ 10.367694] mlx4_core: Initializing 0000:2c:00.2
[ 10.370801] mlx4_core 0000:2c:00.2: enabling device (0000 → 0002)
[ 10.374782] mlx4_core 0000:2c:00.2: Skipping virtual function:2
[ 10.378294] mlx4_core: Initializing 0000:2c:00.3
[ 10.381347] mlx4_core 0000:2c:00.3: enabling device (0000 → 0002)
[ 10.385216] mlx4_core 0000:2c:00.3: Skipping virtual function:3
[ 10.388644] mlx4_core: Initializing 0000:2c:00.4
[ 10.391648] mlx4_core 0000:2c:00.4: enabling device (0000 → 0002)
[ 10.395458] mlx4_core 0000:2c:00.4: Skipping virtual function:4
[ 10.398832] mlx4_core: Initializing 0000:2c:00.5
[ 10.401770] mlx4_core 0000:2c:00.5: enabling device (0000 → 0002)
[ 10.405548] mlx4_core 0000:2c:00.5: Skipping virtual function:5
[ 10.408874] mlx4_core: Initializing 0000:2c:00.6
[ 10.411774] mlx4_core 0000:2c:00.6: enabling device (0000 → 0002)
[ 10.415496] mlx4_core 0000:2c:00.6: Skipping virtual function:6
[ 10.418787] mlx4_core: Initializing 0000:2c:00.7
[ 10.421642] mlx4_core 0000:2c:00.7: enabling device (0000 → 0002)
[ 10.425331] mlx4_core 0000:2c:00.7: Skipping virtual function:7
[ 10.428568] mlx4_core: Initializing 0000:2c:01.0
[ 10.431369] mlx4_core 0000:2c:01.0: Skipping virtual function:8
[ 10.434179] mlx4_core: Initializing 0000:2c:01.1
[ 10.436960] mlx4_core 0000:2c:01.1: Skipping virtual function:9
Hi Fabian,
We support OFED drivers only. Since, you are not using OFED and also the OS that is being used is not Supported, it would be great if you could open a case with OS vendor. Also, I’d like to bring to your notice that the card you are using is End of Life(EOL) . This can be validated from following link —> https://www.mellanox.com/page/eol
Thanks,
Namrata.