ConnectX-3 one-way traffic to switch over SFP+ DAC 10gbps CX311A

[aglatt]

I bought a couple CX311A cards. I updated their firmware to latest 2.42.5000.

I have never been able to ping across this NIC to the switch, much less any other traffic.

When I connect to the switch via Aruba DAC cable SFP+ 10gbps, both sides show UP and 10gbps link speed. The switch shows traffic received, but almost zero “bytes TX” and regularly incrementing “Drops TX”. The server with the NIC shows Sent=0 and Received=64 with pretty much no regular incrementing - just a bit of Received incrementing on first connection. I ran Wireshark on the server, capturing this NIC and it does show the NIC generating some traffic such as ARPs looking for IP destinations and DHCP Discovers, but no received traffic.

My Aruba DAC is in the supported 10gbps cables list in document “Mellanox ConnectX-3 Firmware
Release Notes Rev 2.42.5000” : Aruba HPE J9285B .

I tried both Windows 10 and VMware ESXi 7.0.3 as the OS on the server, with same results.

The server has motherboard Asus Prime X570-P BIOS version 4602, CPU AMD 3950X.

The switch is a Aruba 5406 firmware KB.16.10.0014, and the DAC is connected to a SFP+ port on a J9990A card.

[MvB]

Hello aglatt,

Thank you for posting your inquiry on the NVIDIA Developer Forum - Infrastructure and Networking - Section.

To do the basic triage for this issue, is to eliminate the switch and connect two nodes back to back with the cable.

If the issue resides, replace the cable for another known working cable. Step by step, you can determine if this is a bad cable issue or a faulty adapter.

Thank you and regards,
~NVIDIA Networking Technical Support

[aglatt]

Hi MvB. Thanks for the advice.

I stood up a second client PC with the second CX311A NIC installed and connected directly PC - PC via the two CX311A NICs and the same SFP+ Aruba DAC. The two clients are able to ping each other, no problem, as well as transfer other traffic. So evidently this is some kind of Aruba switch problem.