CX7 link down - Dont neogitate speed to 400G

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi I have two CX7 Connected on two system back to back
Mellanox Technologies MT2910 Family [ConnectX-7] FW 28.43.2026
Product Name: NVIDIA ConnectX-7 adapter card, 400Gb/s NDR IB , Single-port OSFP, PCIe 5.0 x16, Secure boot, No Crypto
Read-only fields:
[PN] Part number: MCX75310AAS-NEAT
[EC] Engineering changes: A9
[V2] Vendor specific: MCX75310AAS-NEAT

connected via OSPF , but they dont negotiate speed and if i manually set to 400G via ethtool links are still down,
it only works in40G Mode ( sudo ethtool -s ens1006np0 speed 40000)

dmesg is flooded with
[71005.536714] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71006.978931] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71016.230408] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71018.237246] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71037.369758] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71040.841875] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71042.869303] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71044.838967] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71057.942795] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71101.217221] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71103.116902] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71111.388193] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71111.767056] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71115.316549] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71005.536714] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71006.978931] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71016.230408] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71018.237246] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71037.369758] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71040.841875] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71042.869303] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71044.838967] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71057.942795] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71101.217221] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71103.116902] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71111.388193] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71111.767056] mlx5_core 0000:85:00.0 ens1006np0: Link down
[71115.316549] mlx5_core 0000:85:00.0 ens1006np0: Link down

Any idea what needs to be done to negotiate speed at 400G

2

Hi @alok.prasad7,

Please verify if your cable is listed in our validated cables list for NDR/400G.

Regards,
Chen

3

@chenh1
Now sure how to check this in the link not exact match, below is the information from ethtool

      AS-5126GS-TNRT2:~$ sudo ethtool -m  ens1006np0
        Identifier                                : 0x19 (OSFP 8X Pluggable Transceiver)
        Power class                               : 1
        Max power                                 : 0.25W
        Connector                                 : 0x23 (No separable connector)
        Cable assembly length                     : 2.00m
        Transmitter technology                    : 0x0a (Copper cable, unequalized)
        Attenuation at 5GHz                       : 0db
        Attenuation at 7GHz                       : 0db
        Attenuation at 12.9GHz                    : 0db
        Attenuation at 25.8GHz                    : 0db
        Vendor name                               : FS
        Vendor OUI                                : 48:b0:2d
        Vendor PN                                 : OSFPFL-400G-PC02
        Vendor rev                                : 00
        Vendor SN                                 : C2506711925-1
        Date code                                 : 250707
        Revision compliance                       : Rev. 4.0
        Module State                              : 0x03 (ModuleReady)
        LowPwrAllowRequestHW                      : Off
        LowPwrRequestSW                           : Off

Settings for ens1006np0:
        Supported ports: [ Backplane ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
                                1000baseKX/Full
                                10000baseKR/Full
                                10000baseR_FEC
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
                                100000baseKR4/Full
                                100000baseSR4/Full
                                100000baseCR4/Full
                                100000baseLR4_ER4/Full
                                50000baseSR2/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
                                10000baseLR/Full
                                10000baseER/Full
                                50000baseKR/Full
                                50000baseSR/Full
                                50000baseCR/Full
                                50000baseLR_ER_FR/Full
                                50000baseDR/Full
                                100000baseKR2/Full
                                100000baseSR2/Full
                                100000baseCR2/Full
                                100000baseLR2_ER2_FR2/Full
                                100000baseDR2/Full
                                200000baseKR4/Full
                                200000baseSR4/Full
                                200000baseLR4_ER4_FR4/Full
                                200000baseDR4/Full
                                200000baseCR4/Full
                                400000baseKR8/Full
                                400000baseSR8/Full
                                400000baseLR8_ER8_FR8/Full
                                400000baseDR8/Full
                                400000baseCR8/Full
                                100000baseKR/Full
                                100000baseSR/Full
                                100000baseLR_ER_FR/Full
                                100000baseCR/Full
                                100000baseDR/Full
                                200000baseKR2/Full
                                200000baseSR2/Full
                                200000baseLR2_ER2_FR2/Full
                                200000baseDR2/Full
                                200000baseCR2/Full
                                400000baseKR4/Full
                                400000baseSR4/Full
                                400000baseLR4_ER4_FR4/Full
                                400000baseDR4/Full
                                400000baseCR4/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None        RS      BASER
        Advertised link modes:  400000baseKR8/Full
                                400000baseSR8/Full
                                400000baseLR8_ER8_FR8/Full
                                400000baseDR8/Full
                                400000baseCR8/Full
                                400000baseKR4/Full
                                400000baseSR4/Full
                                400000baseLR4_ER4_FR4/Full
                                400000baseDR4/Full
                                400000baseCR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Link detected: no (Autoneg, No partner detected)