I installed two VMware ESX Server on the hosts (for example host1 and host2) with ConnectX-3 IB adapters.
After that did these on each nodes (host1 and host2) - I enabled virtualization and Direct I/O in BIOS, added Mellanox device to Toggle device and added Toggle device to each VM.
Each vm with on the Windows Server 2012 R2
I installed VPI drivers from Mellanox site.
Each VM can see Mellanox device, Opensm can see VM’s in the ibnetdiscover but when I try to run ibv_rc_pingpong between VM’s (vm1 located on the host1 and vm2 located on the host2) ibv_devinfo hungs.
You indicated using ConnectX-3 IB so I’m assuming that by asking “Does RDMA work on the ESXi” you probably refer to fully usage of Infiniband/RDMA topology in your Esxi system. if that is the case then the latest Mellanox driver that supports ConnectX-3 family InfiniBand driver & includes the IB kernel-module (mlx4_ib) is the very old mlnx-esxi-VPI-driver v2.4.0.0 (back in 2016/2017…), which is only tested & supported on Esxi5.5 & Esxi6.0, From that point an on - mellanox was not providing IB support within the mlnx_esxi-drivers. Only nmlx4_en & nmlx4_rdma modules were provided for ConnecTx-3 family
If you have not installed IB or VPI driver, then IB/rdma ibv_rc_pingpong will not work