I have Intel Servers with a builtin ConnectX-3, running Linux. I get no link when using a QSFP->SFP+ adapter.

Infrastructure & Networking Adapters and Cables
1

I have Intel Servers with a builtin ConnectX-3. I’m trying to use them with a 10g ethernet switch. I had tested this in a back-to-back connection with a QSFP cable. That worked. Link and connectivity existed.

Now, using a a Mellanox QSFP to SFP+ adapter and a Mellanox SFP I get no link anymore.

There’s no light on the SFP+ (which is tested working) which means I don’t have a switch issue. Since I was able to get link back-to-back I suppose it’s also not a SW issue.

SHOULD it work using the QSFP to SFP+ adapter?

I suppose since this is not a 40g capable adapter it would be wrong to use a 4x10gbit breakout cable.

2

this is what it looks like - i had upgraded the firmware last year, and i suppose on an older HCA this should be OK for all to just work[tm] :-)

myserver # mlxfwmanager

Querying Mellanox devices firmware …

Device #1:

Device Type: ConnectX3

Part Number: ConnectX3-A1-WSP-QDR

Description: ConnectX-3 IB LOM; single-port QSFP; QDR IB (40Gb/s); PCIe3.0 x8 8GT/s; RoHS R6

PSID: INCX-3I358E10201

PCI Device Name: /dev/mst/mt4099_pci_cr0

Port1 MAC: 001e67447c9d

Port2 MAC: 001e67447c9e

Versions: Current Available

FW 2.36.5000 N/A

Status: No matching image found

myserver# lspci -v | grep -i mellanox

01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

Subsystem: Mellanox Technologies Device 0122

01:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

Subsystem: Mellanox Technologies Device 61b0

myserver# mlxconfig -d /dev/mst/mt4099_pciconf0 query

Device #1:

Device type: ConnectX3

PCI device: /dev/mst/mt4099_pciconf0

Configurations: Next Boot

SRIOV_EN True(1)

NUM_OF_VFS 8

LINK_TYPE_P1 ETH(2)

LINK_TYPE_P2 ETH(2)

LOG_BAR_SIZE 3

BOOT_PKEY_P1 0

BOOT_PKEY_P2 0

BOOT_OPTION_ROM_EN_P1 True(1)

BOOT_VLAN_EN_P1 False(0)

BOOT_RETRY_CNT_P1 0

LEGACY_BOOT_PROTOCOL_P1 PXE(1)

BOOT_VLAN_P1 1

BOOT_OPTION_ROM_EN_P2 True(1)

BOOT_VLAN_EN_P2 False(0)

BOOT_RETRY_CNT_P2 0

LEGACY_BOOT_PROTOCOL_P2 PXE(1)

BOOT_VLAN_P2 1

IP_VER_P1 IPv4(0)

IP_VER_P2 IPv4(0)

myserver:~# ethtool eth4

Settings for eth4:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseKX/Full

10000baseKX4/Full

10000baseKR/Full

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: Yes

Advertised link modes: 1000baseKX/Full

10000baseKX4/Full

10000baseKR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: Yes

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000014 (20)

link ifdown

Link detected: no

Here’s kernel info:

myserver # cat /sys/module/mlx4_en/version

4.6-1.0.1

myserver # uname -a

Linux node0106 4.19.68 #1 SMP PREEMPT Tue Aug 27 15:29:53 CEST 2019 x86_64 x86_64 x86_64 GNU/Linux

myserver # rpm -aq |grep -i ofed

mlnx-ofa_kernel-devel-4.6-OFED.4.6.1.0.1.1.ga2cfe08.ol6u9.x86_64

3

Hi Florian,

I suggest to review the supported cables for 10Gb’s

http://www.mellanox.com/pdf/firmware/ConnectX3-FW-2_42_5000-release_notes.pdf

Also i suggest to use latest firmware 2.42.500.

Thanks,

Samer

4

Hi,

we’re ordering two MC2309130-003 cables for testing. I checked, unfortunately fw-ConnectX3-rel-2_36_5000-

ConnectX3-A1-WSP-QDR.bin.zip is the last available on the Mellanox website (https://www.mellanox.com/page/firmware_table_Intel?mtag=oem_firmware_download).

I don’t know what agreements Intel and Mellanox have, but seems the FW maintenance has ended.

I hope that the hybrid cable (as opposed to adapter and sfp+) is gonna do the trick.

Thank you very much for pointing me in the right direction.

5

We received the cables.

With those we get a link up to our 10g switch (as opposed to the QSFP->SFP+ adapter & sfp+.

The next hurdle was absymal performance, between 900 mbit and 2.3gbit/s in summary of 8 streams in iperf2.

Out of a paranoid hunch, I disabled SR-IOV and the 8 virtual functions I had configured.

(mlxconfig … SR-IOV=0 NR_VIFS=1). After this I now get >9gbit/s without even touching the mtu.

It is very frustrating to find this issue but at this point I’m just happy to have finally reached a state with more transmission speed than the 1gbit onboard nics and stable performance.

I’m quite happy to not have needed to replace the intel firmware ;-)

Again, thank you for pointing me at the HCL.