Direct fibre link between ConnectX-4 cards

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

I have 2 hosts with ConnectX-4 LxEN dual port cards and 25Gb fibre modules which I am attempting to connect directly to each other (no switch). However while the “mlxlink” command shows that the module is working perfectly and Rx/Tx values are all good the link does not come up.

However if I connect a fibre between the 2 ports on the same card (i.e. between both ports on one card in the same server) then the link comes up.

I have even tried connecting ports between 2 cards (identical ConnectX-4 cards) in adjacent slots on the same server and the link doesn’t come up. I even used the same fibre which works when I connect both ports on the same card.

I have read through the available documentation a number of times and tried testing with different cards, fibre modules and cables but nothing makes any difference.

Any help very gratefully received.

2

Hi Ian,

I Suggest the following:

Regarding Basic Troubleshooting:

  1. If mlxlink shows no errors, suggest to run basic commands such as:
  • ifconfig up
    • “ibstat” to check the physical/logical port state
    • Try to toggle the link using mlxlink
      • mlxlink -d /dev/mst/mt4115_pciconf0 -a tg
    • Check if the link is up

Thanks,

Samer

3

While Samer’s suggestions are helpful, the problem was finally traced to faulty modules. With fresh modules everything sprang into life.