Disable RoCE and handle RoCE packets as normal UDP packets

[t.barbette]

I’d like to disable RoCE (v2) on the adapter, in a way that packets are not captured by the NIC.

The command “/sys/bus/pci/devices/0000:41:00.0/roce_enable” disables RoCE but packets are still captured. I can verify this using the docker image to capture packets (mentioned in a related thread).

I found the following thread that proposes 2 methods :

RoCEv2 is disabled on ConnectX-5 but the NIC still captures RoCEv2 packets.

Hi. I have a Mellanox ConnectX-5 with OFED 5.4 on Ubuntu 18.04 LTS . I want to parse some RoCEv2 packets with a custom program. So I need the ConnectX-5 not to capture RoCEv2 packets. I’ve followed the article How to Disable RoCE to disable the RoCE. If I get it right, now the RoCEv2 packets should be passed to kernel. But it seems the RoCE is still enabled: ib_send_bw is still working. I used the GID 3 that corresponds to RoCEv2. hping3 UDP test fails. $ hping3 –udp -p 4791 No responding. …

However the first one " sudo mlxconfig -d 41:00.0 s ROCE_CONTROL=1 "
Gives me : Device #1: ---------- Device type: ConnectX5 Name: 09FTMY_071C1T_Ax Description: Mellanox ConnectX-5 Ex Dual Port 100 GbE QSFP Network Adapter Device: 41:00.0 Configurations: Next Boot New -E- The Device doesn’t support ROCE_CONTROL parameter

And the second one use a version of devlink that has a “param” argument that mine does not support:
sudo devlink dev param set pci/0000:82:00.0 name enable_roce value false cmode driverinit Command “param” not found

Any solution?

[hilaryn]

Hello,

Unfortunately, the information provided in this thread will not be sufficient to determine solution.

If you have a current support entitlement contract, please open a case with our support teams for further assistance. If you do not have a current support contract, please reach out to our Contracts team at networking-contracts@nvidia.com to discuss your options.

Thank you,
-Nvidia Network Support

[t.barbette]

I finally found a way to use the devlink solution !

I did this on Cloudlab that has Ubuntu 18.04 by default, for who it may help :)

In short, one needs an updated Kernel, and an updated devlink to be able to disable RoCE

In long, this is the solution on Ubuntu Server 18.04 :

#Install a recent Kernel (5.4 is enough) :
sudo apt-get install --install-recommends linux-generic-hwe-18.04
# Reboot
sudo reboot
# Install an updated iproute2 that contains devlink
wget https://mirrors.edge.kernel.org/pub/linux/utils/net/iproute2/iproute2-5.17.0.tar.xz
tar -Jxvf iproute2-5.17.0.tar.xz
cd iproute2-5.17.0
./configure
make
sudo make install
# And finally you can run the famous command:
sudo devlink dev param set pci/0000:41:00.0 name enable_roce value false cmode driverinit