Switchless SR-IOV config with ConnectX-5 MCX556A-EDAT on Windows Server 2016

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

Hi All,

I am trying to configure a quick setup without switch so I can have my VMs using SR-IOV and talk to each other:

  • Windows Server 2016

  • MCX556A-EDAT x1

  • 100Gbe cable x1

Method:

  1. Connect Port1 to Port2 using the 100Gbe cable directly to form a loop

  2. On Port 1, create a VM Switch with SR-IOV enabled in Hyper-V

  3. Install VM1 and configure to use the VM Switch

  4. Install VM2 and configure to use the VM Switch

Result:

  1. VM1 can connect to host

  2. VM2 can connect to host

  3. Host can connect back to either VM1 or VM2

  4. Confirmed SR-IOV is activated successfully on both VM1 and VM2

Problem:

  1. VM1 cannot connect to VM2

Is there any isolation for VMs if they are on SR-IOV? If I turn SR-IOV off so they fallback to VMQ, they can start connecting to each other without any problem, so it is quite obvious SR-IOV has caused them not communicating.

Or do I have to use a switch to make this setup work?

Cheers

Chris