Problem with vlan with multicast mac over SR-IOV (VMware)

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi,

I have big problem with multicast mac with vlan tag. I suspect filtering and droping packet somewhere within VMware.

Set in VMware VLAN = 4095 for SR-IOV. Tests in both as a mode PCI Device and BYPASS mode (through Port Group).

Problem does not occur when the cards are configured in the passthrough mode (no SR-IOV mode).

Multicasts not tagged, pass without a problem (Precison Time Protocol).

Running tcpdump on the side of the sender log:

Frame 1: 144 bytes on wire (1152 bits), 144 bytes captured (1152 bits)

Encapsulation type: Ethernet (1)

Arrival Time: Mar 4, 2018 21:35:15.204041000 Central European Standard Time

[Time shift for this packet: 0.000000000 seconds]

Epoch Time: 1520195715.204041000 seconds

[Time delta from previous captured frame: 0.000000000 seconds]

[Time delta from previous displayed frame: 0.000000000 seconds]

[Time since reference or first frame: 0.000000000 seconds]

Frame Number: 1

Frame Length: 144 bytes (1152 bits)

Capture Length: 144 bytes (1152 bits)

[Frame is marked: False]

[Frame is ignored: False]

[Protocols in frame: eth:ethertype:vlan:ethertype:sv]

[Coloring Rule Name: Broadcast]

[Coloring Rule String: eth[0] & 1]

Ethernet II, Src: Vmware_51:c4:8e (00:0c:29:51:c4:8e), Dst: Iec-Tc57_04:00:00 (01:0c:cd:04:00:00)

Destination: Iec-Tc57_04:00:00 (01:0c:cd:04:00:00)

Address: Iec-Tc57_04:00:00 (01:0c:cd:04:00:00)

… …0. … … … … = LG bit: Globally unique address (factory default)

… …1 … … … … = IG bit: Group address (multicast/broadcast)

Source: Vmware_51:c4:8e (00:0c:29:51:c4:8e)

Address: Vmware_51:c4:8e (00:0c:29:51:c4:8e)

… …0. … … … … = LG bit: Globally unique address (factory default)

… …0 … … … … = IG bit: Individual address (unicast)

Type: 802.1Q Virtual LAN (0x8100)

802.1Q Virtual LAN, PRI: 7, DEI: 0, ID: 4

  1. … … … = Priority: Network Control (7)

…0 … … … = DEI: Ineligible

… 0000 0000 0100 = ID: 4

Type: IEC 61850/SV (Sampled Value Transmission (0x88ba)

IEC61850 Sampled Values

APPID: 0x4000

Length: 126

Reserved 1: 0x0000 (0)

Reserved 2: 0x0000 (0)

savPdu

noASDU: 1

seqASDU: 1 item

In this document, http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_ESXi_User_Manual_v2.4.0.pdf http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_ESXi_User_Manual_v2.4.0.pdf . Point 3.3.1.5 is written:

“Any vlan-tagged packets sent by the VF are silently dropped.”

Configuration:

Server: 3 x Dell PowerEdge R710 rev II (12 CPU - 24 logical, 96GB RAM)

Mellanox Card: 3 x ConnectX-4 Lx EN-MCX4121A-XCAT (latest FW 14.21.2010, PXE 3.5.0305)

Switches: 2 x Cisco Nexus 3048TP-1GE (latest stable firmware 7.0.3.I7.3)

VMware: DellEMC-ESXi-6.5U1-7388607-A07 (latest from Dell)

VM OS: CentOS Linux release 7.4.1708 (Core, updated)

VMware network driver: nmlx5_core (VMware native, 4.16.10.3-1OEM.650.0.0.4598673)

VM OS network driver mlx5_core: 4.2-1.0.1 (latest Linux driver EN)

How to solve the problem, otherwise research does not make much sense.

I am asking for support, it is very important for Me and My thesis, research.

Best Regards,

Robert

2

How to set or making the VF privileged in VMware ?

Everything works correctly on KVM + Centos.

Settings from the following forum also do not help:

How to turn-off anti-spoofing on a Virtual Func… |VMware Communities How to turn-off anti-spoofing on a Virtual Functio... - VMware Technology Network VMTN

Best Regards,

Robert

3

Robert,

iwork4dell Have you looked into turning off anti-spoofing or making the VF privileged? If you run dmesg on the host are you seeing that the packets are being thrown away because of spoof checks? If you want to send from multiple VLANs on a guest you may not be able to use HW VLAN tagging.

see HowTo Configure MAC Anti-Spoofing for VMs over SR-IOV https://community.mellanox.com/s/article/howto-configure-mac-anti-spoofing-for-vms-over-sr-iov and HowTo Configure Privileged VF on ConnectX-4 https://community.mellanox.com/s/article/howto-configure-privileged-vf-on-connectx-4

4

Sorry I misread your statement that it does work in CentOS just not in VMware.

5

Hi,

I found a newer driver / firmware:

fw-ConnectX4Lx-rel-14_22_1002-MCX4121A-XCA_Ax-UEFI-14.15.19-FlexBoot-3.5.403.bin

nmst-4.9.0.38-1OEM.650.0.0.4598673.x86_64.vib

mft-4.9.0.38-10EM-650.0.0.4598673.x86_64.vib

MLNX-NATIVE-ESX-ConnectX-4-5_4.16.12.12-10EM-650.0.0-7412885.zip

Unfortunately, it did not help, what’s worse I found a bug in SR-IOV, max_vfs is only possible 8 per port (max_vfs = 8,8). The driver does not load when set (max_vfs = 16,16).

In the previous driver, max _vfs was set to 16 per port (max_vfs = 16).

Earlier it was set up globally, one value for all ports. Which allowed to get 32 VFs.

Best Regards,

Robert

6

Problem does not exist on Centos + KVM.

Individual tests on Intel X710 cards on VMware, showed the correct transmission of multicast tagged, but they do not support PTP mode when using VFs (Mellanox can).

Please help to solve the problem.

Best Regards,

Robert