Disconnect problem during MACsec key update using ConnectX-7

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello NVIDIA community,

When a new SA other than the one in use was created during ping communication using MACsec offload in Connext X-7, pinging did not pass.

Is there a way to create a new SA without the communication using the current SA breaking up?

[System Specifications]
・CPU: Intel Xeon Gold 6342, Memory: 256GB
・OS: Ubuntu 22.04, Kernel: 6.8.0-39-generic
・MLNX_OFED: 24.04-0.7.0.0

[Procedure]
1.MACsec encryption settings
Sender

/bin/bash

/opt/mellanox/iproute2/sbin/ip address flush enp23s0np0
/opt/mellanox/iproute2/sbin/ip address add 192.168.247.32/24 dev enp23s0np0
/opt/mellanox/iproute2/sbin/ip link set dev enp23s0np0 up
/opt/mellanox/iproute2/sbin/ip link add link enp23s0np0 macsec0 type macsec sci 1 encrypt on
/opt/mellanox/iproute2/sbin/ip macsec offload macsec0 mac 
/opt/mellanox/iproute2/sbin/ip macsec add macsec0 tx sa 0 pn 1 on key 00 dffafc8d7b9a43d5b9a3dfbbf6a30c16
/opt/mellanox/iproute2/sbin/ip macsec add macsec0 rx sci 1 on
/opt/mellanox/iproute2/sbin/ip macsec add macsec0 rx sci 1 sa 0 pn 1 on key 00 ead3664f508eb06c40ac7104cdae4ce5
/opt/mellanox/iproute2/sbin/ip address flush macsec0
/opt/mellanox/iproute2/sbin/ip address add 192.168.249.32/24 dev macsec0
/opt/mellanox/iproute2/sbin/ip link set dev macsec0 up

Receiver

#!/bin/bash

/opt/mellanox/iproute2/sbin/ip address flush enp23s0np0
/opt/mellanox/iproute2/sbin/ip address add 192.168.247.33/24 dev enp23s0np0
/opt/mellanox/iproute2/sbin/ip link set dev enp23s0np0 up
/opt/mellanox/iproute2/sbin/ip link add link enp23s0np0 macsec0 type macsec sci 1 encrypt on
/opt/mellanox/iproute2/sbin/ip macsec offload macsec0 mac 
/opt/mellanox/iproute2/sbin/ip macsec add macsec0 tx sa 0 pn 1 on key 00 ead3664f508eb06c40ac7104cdae4ce5
/opt/mellanox/iproute2/sbin/ip macsec add macsec0 rx sci 1 on
/opt/mellanox/iproute2/sbin/ip macsec add macsec0 rx sci 1 sa 0 pn 1 on key 00 dffafc8d7b9a43d5b9a3dfbbf6a30c16
/opt/mellanox/iproute2/sbin/ip address flush macsec0
/opt/mellanox/iproute2/sbin/ip address add 192.168.249.33/24 dev macsec0
/opt/mellanox/iproute2/sbin/ip link set dev macsec0 up
  1. ping
#!/bin/bash

ping 192.168.249.32(example)

3.Create new SA
Sender

#!/bin/bash

/opt/mellanox/iproute2/sbin/ip macsec add macsec0 tx sa 1 pn 1 on key 00 dffafc8d7b9a43d5b9a3dfbbf6a30c17

Receiver

#!/bin/bash

/opt/mellanox/iproute2/sbin/ip macsec add macsec0 rx sci 1 sa 1 pn 1 on key 00 dffafc8d7b9a43d5b9a3dfbbf6a30c17
  • No ping response at the time of new SA creation (TX side).
  • When the packet is captured, the PN is reset when the new SA is created in sender, and the SA (AN) is sent as it was.

4.SA Switching

#!/bin/bash

/opt/mellanox/iproute2/sbin/ip link set macsec0 type macsec encodingsa 1
  • After switching to the new SA, ping communication resumed.
  • When the packet is captured, the AN is also switched to the new SA settings and sent.
    (Without MACsec offload, the ping continued to communicate when running 3 above, and after switching 4, it communicated without any problems.)

Thank you in advance for your assistance!

2

No way do this.

3

Thank you for your reply.

Do you mean that MACsec key update without communication interruption is impossible?