We have a 2x100G NIC from Mellanox, but we haven’t been able to establish a link against our 100G switch ports. Our test scenario is very simple. It looks like this:
PC Mellanox 100G port <-------> 100G Direct Attach Copper Cable <-------> 100G switch port
PC details:
- Ubuntu 18.04.1 installed (kernel 4.15.0-29-generic)
- Mellanox part number MCX416A-CCAT
- Mellanox driver installed from tarball:
- First installed from mlnx-en-4.4-2.0.7.0-ubuntu18.04-x86_64.tar.tgz
- Upgraded to mlnx-en-4.5-1.0.1.0-ubuntu18.04-x86_64.tgz
- NIC details:
- PSID: MT_2150110033
- UEFI 14.17.0011
- PCI Device Name: 01:00.0
- Base GUID: 506b4b0300284944
- FW 12.24.1000
- PXE 3.5.0603
Switch port details:
- Interface mode CAUI4
- Speed/Duplex:
- Forced 100G Full-Duplex
- We have also tried autonegotiation advertising only 100G full
I have read in the NIC manual that that autoneg is never really off… Is there a way to really force the interface to operate in a specific mode?
Also, are there any extra debug options and logs that we could activate in Linux side to get more information about the possible autonegotiation problem?
Additional information: We have confirmed that connecting both Mellanox 100G ports to each other works (100G link up); and we have also confirmed that connecting two switch ports at 100G with/without autoneg works. This was all done with the same cables as used in the tests above. Therefore, we don’t think there is a hardware problem! Perhaps it is a (media/interface type) negotiation problem?
Thanks in advance for any information!