No Ping with ConnectX-5 VPI Socket Direct on ESXi 7.0 U2

[it1109]

Hi guys,

I have 2 servers interconnecting switchless with ConnectX-5 VPI 100Gb ETH/IB Socker-Direct (2x8 PCIe), all well configured and I can also ping between Linux VMs (OFED Linux driver) from both hosts (via PCI passthrough the nic).
But when toggling off the passthrough mode from esxi 7.0U2 (vsphere / vcenter 7 U3) I cannot ping between my vSS. I dont unterstand it, because the same vSS work with other nics, mtu is set on 1500 and Im confused about esxi status: link up but no ping between hosts possible?! I tried to ping vmk and other maschines from the second host. I used 4.21.x driver and 4.19.x ethernet drivers… i also tried with OFED (but it is not supported on esxi 7).

does anyone have experience how to connect these two cards in VMware environment?

thanks!

[ipavis]

Hello,

Are you using the VMware inbox driver or the Mellanox\NVIDIA drivers?
Please provide the full version of the drivers you are using.

Can you please provide the output for this command by running from the ESXi CLI

esxcli network nic list

Thanks,
Ilan.

[it1109]

Hello @ipavis ,

im using inbox and now try going back to async 4.21.71.1/101 with firmware 16.19.1016 (also tried 16.32.x)

this is my output:
[root@esxi01:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description

---

vmnic0 0000:18:00.0 ixgben Up Up 10000 Full 0c:c4:7a:f3:ef:02 1500 Intel(R) 82599 10 Gigabit Dual Port Network Connection
vmnic1 0000:18:00.1 ixgben Up Down 0 Half 0c:c4:7a:f3:ef:03 1500 Intel(R) 82599 10 Gigabit Dual Port Network Connection
vmnic2 0000:af:00.0 bnxtnet Up Up 10000 Full bc:97:e1:2a:2e:d0 1500 Broadcom BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller
vmnic3 0000:af:00.1 bnxtnet Up Up 10000 Full bc:97:e1:2a:2e:d1 1500 Broadcom BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller
vmnic4 0000:b0:00.0 bnxtnet Up Up 10000 Full bc:97:e1:2a:29:00 1500 Broadcom BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller
vmnic5 0000:b0:00.1 bnxtnet Up Up 10000 Full bc:97:e1:2a:29:01 1500 Broadcom BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller
vmnic6 0000:86:00.0 nmlx5_core Up Down 0 Half f4:52:14:22:01:80 1500 Mellanox Technologies ConnectX-5 VPI adapter card with Multi-Host Socket Direct supporting dual-socket server EDR IB (100Gb/s) and 100GbE dual-port QSFP28 2x PCIe3.0 x8 35cm harness (MCX556M-ECAT-S35A)
vmnic7 0000:86:00.1 nmlx5_core Up Up 100000 Full f4:52:14:22:01:81 9000 Mellanox Technologies ConnectX-5 VPI adapter card with Multi-Host Socket Direct supporting dual-socket server EDR IB (100Gb/s) and 100GbE dual-port QSFP28 2x PCIe3.0 x8 35cm harness (MCX556M-ECAT-S35A)
vmnic8 0000:3c:00.0 nmlx5_core Up Down 0 Half f4:52:14:22:01:86 1500 Mellanox Technologies ConnectX-5 VPI adapter card with Multi-Host Socket Direct supporting dual-socket server EDR IB (100Gb/s) and 100GbE dual-port QSFP28 2x PCIe3.0 x8 35cm harness (MCX556M-ECAT-S35A)
vmnic9 0000:3c:00.1 nmlx5_core Up Up 100000 Full f4:52:14:22:01:87 9000 Mellanox Technologies ConnectX-5 VPI adapter card with Multi-Host Socket Direct supporting dual-socket server EDR IB (100Gb/s) and 100GbE dual-port QSFP28 2x PCIe3.0 x8 35cm harness (MCX556M-ECAT-S35A)