RDMA Setup in ESXi does not seem to be working

Infrastructure & Networking Adapters And Cables
#1

Hi,

I am trying to setup RoCE on ESXi to get a feel for it. Unfortunatly it doesnt seem to work yet and I am not sure what I’ve beend doing wrong:

Test ESXi boxes (6.7U3) have CX3’s (non pro so RoCE v1 only)

followed https://community.mellanox.com/s/article/howto-run-roce-over-l2-enabled-with-pfc–esxi-x

to enable on ESXi

Basically on each box I did

esxcli system module parameters set -m nmlx4_en -p “pfctx=0x08 pfcrx=0x08”

esxcli system module parameters set -m nmlx4_rdma -p “pcp_force=3”

reboot

On the switch I followed

https://community.mellanox.com/s/article/howto-enable-pfc-on-mellanox-switches–switchx-x

interface ethernet 1/1-1/6 flowcontrol send off force

interface ethernet 1/1-1/6 flowcontrol receive off force

dcb priority-flow-control enable

dcb priority-flow-control priority 3 enable

interface ethernet 1/1-1/6 dcb priority-flow-control mode on force

I didnt modify my current VLAN settings as I have a bunch set up for different ESXi traffic already.

On ESXi there was no mapping mentioned of VLAN to priority class (on Linux it is mapped with vconfig per VLAN → https://community.mellanox.com/s/article/howto-run-roce-and-tcp-over-l2-enabled-with-pfc), so I didnt do anything else.

Networking is still working fine, no errors, but no RoCE seems to be in use - I can’t see a single pause frame…

So anyone any idea what I am missing?

Could it be that regular ESXi traffic will not generate RDMA traffice?

I had hoped from the description

"

RDMA in ESXi

  • RDMA can be useful to accelerate many of the hypervisor services including; SMP-FT, NFS and iSCSI.

"

that it might run in native ESXi too when enabled on interface level.

Only thing I tested till now were some vMotions…

Thanks

#2

The configuration that is described in the article should work fine, as long as you install & use VMware inbox driver “ESXi6.7-v3.17.9.12-CX3-is-WMware-Inbox-driver”. this is the only driver that supports Mellanox Connectx-3 w/Esxi6.7

Download it from VMware site, try re-configuring Roce over L2 with PFC and see if it works

#3

Hi Avi,

thanks for that missing detail; I will give it a try but it might take awhile since I have repurposed the environment in the meantime since I could not get it to run.

Could you elaborate on the funcitonality within VMWare that will actually be using RDMA traffic if configured correctly? I.e. for what kind of activity would I actually be seeing RDMA traffic being passed through the switch?

Thanks