Trouble getting CX416A 100G link up

We have a 2x100G NIC from Mellanox, but we haven’t been able to establish a link against our 100G switch ports. Our test scenario is very simple. It looks like this:

PC Mellanox 100G port <-------> 100G Direct Attach Copper Cable <-------> 100G switch port

PC details:

  • Ubuntu 18.04.1 installed (kernel 4.15.0-29-generic)
  • Mellanox part number MCX416A-CCAT
  • Mellanox driver installed from tarball:
    • First installed from mlnx-en-4.4-
    • Upgraded to mlnx-en-4.5-
  • NIC details:
    • PSID: MT_2150110033
    • UEFI 14.17.0011
    • PCI Device Name: 01:00.0
    • Base GUID: 506b4b0300284944
    • FW 12.24.1000
    • PXE 3.5.0603

Switch port details:

  • Interface mode CAUI4
  • Speed/Duplex:
    • Forced 100G Full-Duplex
    • We have also tried autonegotiation advertising only 100G full

I have read in the NIC manual that that autoneg is never really off… Is there a way to really force the interface to operate in a specific mode?

Also, are there any extra debug options and logs that we could activate in Linux side to get more information about the possible autonegotiation problem?

Additional information: We have confirmed that connecting both Mellanox 100G ports to each other works (100G link up); and we have also confirmed that connecting two switch ports at 100G with/without autoneg works. This was all done with the same cables as used in the tests above. Therefore, we don’t think there is a hardware problem! Perhaps it is a (media/interface type) negotiation problem?

Thanks in advance for any information!

Hi Felipe,

It would be great if you could let me know the P/N of the switch and cable? Also, Can you please let me know what is the FEC configured on switch? In additional, please provide the output by running following command:

  1. #mst status (To get mst device name)
  2. #mlxlink -d -emc



Hi Felipe,

I see you have also opened a support case with us for the same issue. I will continue working with you via that case and close this.