Disable RoCE and handle RoCE packets as normal UDP packets

I’d like to disable RoCE (v2) on the adapter, in a way that packets are not captured by the NIC.

The command “/sys/bus/pci/devices/0000:41:00.0/roce_enable” disables RoCE but packets are still captured. I can verify this using the docker image to capture packets (mentioned in a related thread).

I found the following thread that proposes 2 methods :

However the first one " sudo mlxconfig -d 41:00.0 s ROCE_CONTROL=1 "
Gives me : Device #1: ---------- Device type: ConnectX5 Name: 09FTMY_071C1T_Ax Description: Mellanox ConnectX-5 Ex Dual Port 100 GbE QSFP Network Adapter Device: 41:00.0 Configurations: Next Boot New -E- The Device doesn’t support ROCE_CONTROL parameter

And the second one use a version of devlink that has a “param” argument that mine does not support:
sudo devlink dev param set pci/0000:82:00.0 name enable_roce value false cmode driverinit Command “param” not found

Any solution?


Unfortunately, the information provided in this thread will not be sufficient to determine solution.

If you have a current support entitlement contract, please open a case with our support teams for further assistance. If you do not have a current support contract, please reach out to our Contracts team at networking-contracts@nvidia.com to discuss your options.

Thank you,
-Nvidia Network Support

I finally found a way to use the devlink solution !

I did this on Cloudlab that has Ubuntu 18.04 by default, for who it may help :)

In short, one needs an updated Kernel, and an updated devlink to be able to disable RoCE

In long, this is the solution on Ubuntu Server 18.04 :

#Install a recent Kernel (5.4 is enough) :
sudo apt-get install --install-recommends linux-generic-hwe-18.04
# Reboot
sudo reboot
# Install an updated iproute2 that contains devlink
wget https://mirrors.edge.kernel.org/pub/linux/utils/net/iproute2/iproute2-5.17.0.tar.xz
tar -Jxvf iproute2-5.17.0.tar.xz
cd iproute2-5.17.0
sudo make install
# And finally you can run the famous command:
sudo devlink dev param set pci/0000:41:00.0 name enable_roce value false cmode driverinit

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.