supporting 1000base with a connectx-3

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello all,

I am still learning the ropes with these cards so hopefully I include the right details.

the card in this server is:

# mstflint -d 42:00.0 q

Image type: FS2

FW Version: 2.32.5100

FW Release Date: 3.9.2014

Product Version: 02.32.51.00

Rom Info: type=PXE version=3.4.306 devid=4099 proto=ETH

Device ID: 4099

Description: Node Port1 Port2 Sys image

GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff

MACs: f452141fd030 f452141fd031

VSD:

PSID: MT_1080120023

I can see the following with ethtool:

# ethtool p4p1

Settings for p4p1:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseKX/Full

10000baseKR/Full

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: No

Advertised link modes: 1000baseKX/Full

10000baseKR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000014 (20)

link ifdown

Link detected: no

The network engineer on the Cisco side says he can see the SFP on that side is happy but no link. I can’t see a link either and now the question is what have I done wrong in relation to getting this card working. The drivers for the card appear to be installed correctly etc.

And yet speed and duplex are unknown. The SFPs on the switch side can only support 1000.

Thanks for your time all.

2

Hi,

I’m using an LC\LC fibre cable. The SFP in the card is an MLX.MFM1T02A-SR (part number I have on invoice) and the SFP on the Cisco side is a Cisco module - the part number of which I do not have to hand.

OFED self test shows the cards as not connected.

# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected … 1

PCI Device Check … PASS

Kernel Arch … x86_64

Host Driver Version … MLNX_OFED_LINUX-2.3-2.0.0 (OFED-2.3-2.0.0): 2.6.32-504.el6.x86_64

Host Driver RPM Check … PASS

Firmware on CA #0 NIC … v2.32.5100

Firmware Check on CA #0 (NIC) … PASS

Host Driver Initialization … PASS

Number of CA Ports Active … 0

Port State of Port #1 on CA #0 (NIC)… DOWN (Ethernet)

Port State of Port #2 on CA #0 (NIC)… DOWN (Ethernet)

Error Counter Check on CA #0 (NIC)… NA (Eth ports)

Kernel Syslog Check … PASS

Node GUID on CA #0 (NIC) …

------------------ DONE ---------------------

I’m just trying to figure our where in the chain things are not as they should be…

3

mmmm

I think you are going to say I need an MC3208011-SX SFP to get this to work.

4

What cable / SFP adapter are you using ? do you have OFED installed ?