Hello,
There is an environment with linux based servers, and there is some repeating behavior on different nodes with the following network adapters:
Mellanox MT27800 Family [ConnectX-5] - 16.35.2000 (MT_0000000012 == MCX516A-CCA)
Mellanox MT2892 Family [ConnectX-6 Dx] - 22.43.1014 (MT_0000000436 == MCX623106AC-CDA)
Each linux server has 4 hics, connected to Cisco ACI switches with VPC configured, and some vlans over them – two hics goes to switch A, and the other two hics goes to switch B, and all the four ports participates with the same LACP group.
The behavior is that one of the nic pairs - 2 hics out of the 4 – which goes to the same switch, has errors – “rx_crc_errors_phy”, (seen when running “ethtool -S | grep rx_crc”).
on these hics, “rx_jabbers_phy” and equal value of “rx_oversize_pkts_phy” has been observed as well.
They are not necessarily connected to a specific switch, one server can have the issue with ports connected to one switch, and another server can have the issue with ports connected to another switch.
Everything was replaced to troubleshoot – SFPs, convertors, cables, new ports in the switch, ports crossing – but always we see that the ports connected to one of the switches are reporting the errors, although it doesn’t see like a physical issue.
From the switches side, no errors are seen on the ports.
Network ports from the switch are set to MTU of 9000, but it happens on linux servers with either 1500 or 9000 MTU configuration on the hics.
Ant ideas regarding the causes and a solution?
A few points stand out:
The issue does not follow a particular switch.
If there were a defect in a specific leaf, ASIC, line card, or ACI software process, you would expect the errors to consistently follow that device. Instead, different servers experience the problem on different leaf switches. The only common pattern is that the affected interfaces are always the pair connected to the same leaf. That makes a single-switch hardware or software defect much less likely.
The errors are being detected by the NIC, not the switch.
The counters being incremented are rx_crc_errors_phy, rx_jabbers_phy, and rx_oversize_pkts_phy on the Mellanox adapters. These are PHY-level receive counters maintained by the NIC hardware itself.
On the switch side, no CRC, giant, alignment, or other physical-layer errors are reported. If the switch were transmitting corrupted frames, it would be unusual to see absolutely no corresponding evidence on the switch interfaces while multiple hosts report receive-side PHY errors.
No widely known ACI defect matches these symptoms.
Given all of the above, I would focus the investigation on the host side and the host-to-switch interoperability layer:
Verify driver versions (mlx5_core) and compare affected versus unaffected hosts.
Test newer Mellanox/NVIDIA firmware versions if available.
Perform packet captures and correlate timestamps with counter increments.
Open a case with NVIDIA, providing the PHY counters, firmware versions, and switch-side statistics.
Hello,
In wireshark - out of order packets are seen, comparing to the nics, while they are not seen on the nics without the errors.
Regarding the ACI , there is one implementation with NX-OS version, also there it’s LACP of 4 ports, two nics in each switch, and there are no nic errors with this configuration, so we thought it may be a combination of nics and switch configuration.