CX4 links at 40G only - was "Interoperability of CX4 with SONIC switch"

Infrastructure & Networking Adapters And Cables
#21

@Martijn van Breugel​ I have now upgraded to fw-ConnectX4-rel-12_25_1020-MCX456A-ECA_Ax-UEFI-14.18.19-FlexBoot-3.5.701 which I assume is the fw that you were referring to, but there is no change:

Card to self - 100G (with C & E cable)

Card to other Card (CX5) - 100G (with C & E cable)

Card to Switch - no connect unless switch set to 40G

CX5 to switch -100G (with C & E cable)

Do you have any other ideas?

If this is not the card but the switch (which seems a possibility given the above results) - would you think this is a hardware or a software issue?

Hardware would be unsolvable for me, software I could try to work on. Otoh, it happens with both OcNos and SoNIC, so also hard to understand…

#22

Hi Thomas,

Sorry for the late reply. I was on vacation the last 2 weeks.

It seems that it becomes an interoperability issue between the ConnectX-4 VPI and the switch.

I think the best thing to do is to open a support case directly with Mellanox Technical Support, so we can continue to debug on a deeper level.

As you are based in EMEA, our EMEA colleagues will assist you but I will be following the case.

Many thanks,

~Martijn

#23

Hi @Martijn van Breugel​ ,

somehow notifications are not working as expected:/

I took the liberty to reopen the original case - can you please re-assign appropriately?

Many thanks for your help,

best regards,

Thomas

#24

I finally managed to resolve this issue this week.

It turns out that CX4’s (and Chelsio’s) don’t seem to be able to properly auto-negotiate FEC settings on Sonic (and OcNos) switches [or maybe its the Broadcom or Celestica HW].

When manually configuring FEC to setting RS (instead of none or FC) the CX4’s linked up with all cables types I tried (Mellanox C and E types and also Dell branded ones).