IPSec RDMA using ConnectX-6 DX

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

I was able to offload IPsec through both strongswan and ip xfrm. I can observe ESP frames through tcpdump in both approaches with iperf and jumbo frames. However:

  • there is a significant performance impact with the offload, Is this expected? any way to tune this further?
  • RoCE V2 does not seem to get encrypted after setting IPsec on the NIC. Is there any extra step to getting RDMA to use IPsec?

Thanks!

2

Hello, anyone can answer?

3

Hi Sunil,

Can you provide more details on your environment? Are you using OFED or Upstream kernel drivers? If upstream which kernel version are you using? Is this IPSec Full Offload? We had a bug with IPSec offload in our kernel driver which was fixed upstream in kernel 6.15. Its been merged to OFED 25.07 which hasn’t been released yet. This is the patch series: Making sure you're not a bot!

Regards,
Kiran

4

Hi Kiran,
Thanks for checking. Here’s my environment:

  • debian 12 with upstream linux 6.1.37
  • all tools and utilities are from debian 12 except:
    • tcpdump which I recompiled from latest version to examine RoCE frames
    • I have tried recompiling strongswan bits from BF-5.9.10, but it does not understand offload=full.

I don’t know how else to use ‘IPsec Full offload’ since none of those config files exist on the system.
I have also tried DOCA /opt/melanox/iproute2/sbin/ip xfrm but it does not help RoCE traffic.

Thanks!!

5

So no OFED based on your reply. Correct me if I am wrong. If no OFED, can you please move to kernel 6.15 or later (which has the patches to fix Rx Offload failure) and give it a try?

6

while I don’t understand what you mean by ‘no OFED’, there are very limited mellanox tools installed. I can try the OFED path if you can provide instructions (but would prefer not to update kernel, only drivers).

It will be hard for me to install the 6.15 kernel. The closest I can try out is trixie - 6.12.35.

7

For Mellanox Drivers, you can use either the Inbox Drivers (whatever comes by default with the kernel you installed) or explicitly install OFED (which also packages Mellanox Drivers along with several other tools such as mst tools, perf tools, etc). If you want to install OFED you can give this a try.

  1. Download latest LTS OFED ( MLNX_OFED_LINUX-24.10-3.2.5.0) from here (as per your OS Distro): Linux InfiniBand Drivers In your case I guess it would be Debian 12.1 or Debian 12.5 depending on which one you use. If you use something different that these, I am afraid we don’t have a OFED Package for that version.

  2. Once you untar the package you can install OFED drivers for your custom kernel 6.1.37 as follows: ./mlnxofedinstall --force --with-kernel-support

Follow the instructions to complete the installation if it complains about any missing packages, etc

  1. Once the default OFED Drivers are installed you will need to patch the drivers and reload them. To do that extract the OFED Driver sources from the OFED tarball you downloaded under src directory

tar xzvf MLNX_OFED_SRC-24.10-3.2.5.0.tgz

cd MLNX_OFED_SRC-24.10-3.2.5.0/

cd SOURCES/

tar xzvf mlnx-ofed-kernel_24.10.OFED.24.10.3.2.5.1.orig.tar.gz

cd mlnx-ofed-kernel_24.10.OFED.24.10.3.2.5.1/

./configure --all

Copy patches shared on google drive (ipsec.tgz) and untar it. Apply patches

cat ipsec/0001-net-mlx5e-Add-helper-function-to-update-IPSec-defaul.patch | patch -p1

cat ipsec/0002-net-mlx5e-Change-the-destination-of-IPSec-RX-SA-miss.patch | patch -p1

cat ipsec/0003-net-mlx5e-Add-correct-match-to-check-IPSec-syndromes.patch | patch -p1

cat ipsec/0004-net-mlx5e-Move-IPSec-policy-check-after-decryption.patch | patch -p1

cat ipsec/0005-net-mlx5e-Skip-IPSec-RX-policy-check-for-crypto-offl.patch | patch -p1

cat ipsec/0006-net-mlx5e-Add-num_reserved_entries-param-for-ipsec_f.patch | patch -p1

cat ipsec/0007-net-mlx5e-Add-pass-flow-group-for-IPSec-RX-status-ta.patch | patch -p1

cat ipsec/0008-net-mlx5e-Support-RX-xfrm-state-selector-s-UPSPEC-fo.patch | patch -p1

Build updated drivers with patches

make -j8

make install_kernel

Run openibd restart to load the newly build drivers

/etc/init.d/openibd restart