We are experiencing an issue with MACsec hardware offloading (offload mac) on ConnectX-8 where the receiver consistently drops incoming traffic after a fixed number of packets are processed on the active SA.
Depending on the cipher suite used, the failure occurs at the following thresholds:
- With GCM-AES-XPN-256, the failure occurs at approximately: 5,368,709,120 packets = 5 × 2³⁰
- With GCM-AES-256, it occurs at approximately: 3,221,225,472 packets = 3 × 2³⁰
At the threshold, the receiver drops traffic and starts reporting macsec_rx_pkts_drop via ethtool. No corresponding devlink health or dmesg errors are reported. Manually rotating to a new MACsec SA before or after the threshold prevents/recovers from the failure.
Ethtool Output at Failure (XPN-256):
# ethtool -S eth1 | grep macsec
macsec_rx_pkts: 5368709112
macsec_rx_bytes: 7954261605643
macsec_rx_pkts_drop: 38407
macsec_rx_bytes_drop: 58116571
macsec_tx_pkts: 58994442
macsec_tx_bytes: 5784385364
macsec_tx_pkts_drop: 0
macsec_tx_bytes_drop: 0
Environment:
- OS: Linux Kernel 6.18.41
- Driver:
mlx5_core - Device: NVIDIA ConnectX-8 SuperNIC (400GbE)
- Firmware: 40.50.1002 (FS5)
Firmware Details (mstflint):
# mstflint -d 41:00.0 query full
Image type: FS5
FW Version: 40.50.1002
FW Release Date: 18.8.2026
Part Number: 900-9X81Q-00CN-ST0_Ax
Description: NVIDIA ConnectX-8 C8240 HHHL SuperNIC; 400GbE (default) / 400Gb/s IB; Dual-port QSFP112; PCIe 6 x16 with x16 PCIe Socket Direct Extension option; Crypto Enabled; Secure Boot Enabled
Product Version: 40.50.1002
Rom Info: type=UEFI version=14.43.10 cpu=AMD64,AARCH64
type=PXE version=3.9.101 cpu=AMD64
...
Allocated GUID: 0000000000000000
Image VSD: N/A
Device VSD: N/A
PSID: MT_0000001222
Security Attributes: secure-fw
Default Update Method: fw_ctrl
Life cycle: GA SECURED
Secure Boot Capable: Enabled
EFUSE Security Ver: 0
Image Security Ver: 0
Security Ver Program: Manually ; Disabled
Encryption: Enabled
PCIe switch mode only: Disabled
Steps to Reproduce:
Setup Host 1 (TX):
ip link add link eth0 macsec0 type macsec \
port 1 encrypt on cipher gcm-aes-xpn-256 offload mac
ip macsec add macsec0 tx sa 0 xpn 1 \
ssci 123 salt d3de86a01494e340a1f542f6 \
on key 01 d29a43c8cba96a325f6b6a40a214c58cd29a43c8cba96a325f6b6a40a214c58c
ip macsec add macsec0 rx port 1 address e8:9e:49:38:1d:59
ip macsec add macsec0 rx port 1 address e8:9e:49:38:1d:59 sa 0 xpn 1 \
ssci 123 salt e53978f58ec409c80ec7da1f \
on key 02 a1e15a1d91222196fde87b2d75a4fac0a1e15a1d91222196fde87b2d75a4fac0
ip addr add 10.201.198.1/24 dev macsec0
ip link set macsec0 up
Setup Host 2 (RX):
ip link add link eth1 macsec0 type macsec \
port 1 encrypt on cipher gcm-aes-xpn-256 offload mac
ip macsec add macsec0 tx sa 0 xpn 1 \
ssci 123 salt e53978f58ec409c80ec7da1f \
on key 02 a1e15a1d91222196fde87b2d75a4fac0a1e15a1d91222196fde87b2d75a4fac0
ip macsec add macsec0 rx port 1 address a4:a6:4e:3d:62:10
ip macsec add macsec0 rx port 1 address a4:a6:4e:3d:62:10 sa 0 xpn 1 \
ssci 123 salt d3de86a01494e340a1f542f6 \
on key 01 d29a43c8cba96a325f6b6a40a214c58cd29a43c8cba96a325f6b6a40a214c58c
ip addr add 10.201.198.2/24 dev macsec0
ip link set macsec0 up
Send traffic from Host 1 to Host 2. Once the PN hits the thresholds outlined above, macsec_rx_pkts_drop rapidly increases and traffic halts.