Vsan rdma question

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Im using vsan setup with rdma enabled.
I use connect x-6 dual port 100 g nics.

From the switch end, i have changed the max mtu as 9000,however, from the card, i see is max 4096(its printed on the card by default) why its showing like that?

Note, i use DVS from vcenter where i have already set mtu as 9000.

For rdma, previously it was showing as cee mode and tried to change it to ieee mode but no luck from esxi end.

Now it says mode is invalid/unknown.

Could you please help?

2

Hi,

Thanks for your question.
The maximum possible MTU for RDMA is 4k.
Please adjust it to 4k in all the relevant places in your RDMA environment.

Best Regards,
Anatoly

3

Hi Anatoly,
Happy new year 2023 and thanks for the response.

I have few queries. Could you please help with this?

  1. We use VSAN 8.0 and VSAN traffic goes via 2 * 100 G NIC.“ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AN-CDA)”.
    We recently changed the mode from CEE to IEEE as per this article.After changing,now the RDMA mode is showing as “unknown”. I dont know why,My current firmware version of 100G NIC is “22.32.10.14”

2.If we disable the RDMA on 100G NIC,then the NIC will support 9000 MTU ?

3.Do we need to upgrade the latest firmware version? If yes,please suggest the version no for VSAN 8.0

Thanks,
Raj

4

Could you please update me the correct to change the RDMA mode in to IEEE?

I used these command,but still it says invalid/unknown from ESXi/VSAN end/

/opt/mellanox/bin/mlxconfig -d mt4117_pciconf0 set LLDP_NB_DCBX_P1=1 LLDP_NB_RX_MODE_P1=2 LLDP_NB_TX_MODE_P1=2 LLDP_NB_DCBX_P2=1 LLDP_NB_RX_MODE_P2=2 LLDP_NB_TX_MODE_P2=2 DCBX_WILLING_P1=1 DCBX_IEEE_P1=1 DCBX_CEE_P1=0 DCBX_WILLING_P2=1 DCBX_IEEE_P2=1 DCBX_CEE_P2=0

Thanks
raj