Mellanox connectX 3 and connectX 4 Loop or crossover connection

Hello,

I have two Mellanox card, with two Supermicro servers with Ubuntu Server 18.04.3, I am trying to connect them as crossover with qspf+ 40Gb cable, but I got only 10Gb connections.

at the same time when I am connecting both cards to Cisco switch I am getting 40Gb connections.

Also as loop (port to port) at the same card I can connect ConnectX3 but again getting only 10Gb connection.

But when I am trying to connect ConnectX4 NEtwork is not working at all, and at syslog I getting errors as [mlx5 core cable error, unknown identifier]

And difference in ethtool report

With connection to Cisco Switch:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseKX/Full

10000baseKX4/Full

10000baseKR/Full

40000baseCR4/Full

40000baseSR4/Full

56000baseCR4/Full

56000baseSR4/Full

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: Yes

Supported FEC modes: Not reported

Advertised link modes: 1000baseKX/Full

10000baseKX4/Full

10000baseKR/Full

40000baseCR4/Full

40000baseSR4/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: Yes

Advertised FEC modes: Not reported

Link partner advertised link modes: 40000baseCR4/Full

Link partner advertised pause frame use: No

Link partner advertised auto-negotiation: Yes

Link partner advertised FEC modes: Not reported

Speed: 40000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000014 (20)

link ifdown

Link detected: yes

With Connection as crossover ConnectX3 to ConnectX4

Supported ports: [ FIBRE ]

Supported link modes: 1000baseKX/Full

10000baseKX4/Full

10000baseKR/Full

40000baseCR4/Full

40000baseSR4/Full

56000baseCR4/Full

56000baseSR4/Full

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: Yes

Supported FEC modes: Not reported

Advertised link modes: 1000baseKX/Full

10000baseKX4/Full

10000baseKR/Full

40000baseCR4/Full

40000baseSR4/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: Yes

Advertised FEC modes: Not reported

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000014 (20)

link ifdown

Link detected: yes

Can I emulate this (Link partner advertised link modes: 40000baseCR4/Full ) while connecting as crossover??

Thank you very much

Please, check that you are using supported cable listed in ConnectX-3 and ConnectX-4 latest release notes - most important. For not listed cables, behaviour might be unexpected.

Verify that you are using latest available firmware for both cards.

Try to set fixed speed and/or disable autonegotiation using ethtool.

Please, note that firmware for ConnectX-3 HCA is a final and no additional feature will be added to it.