ConnectX-3 10Gb ok but not 40Gb (Ubuntu 16.04)

Ubuntu server 16.04.02 fresh install on a SuperMicro 7048GR-TR

ConnectX-3 dual port board.

It is recognised fine:


lspci -v | grep -i mellan -A 10 82:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3] Physical Slot: 21 Flags: bus master, fast devsel, latency 0, IRQ 35 Memory at fb100000 (64-bit, non-prefetchable) [size=1M] Memory at 387ff2800000 (64-bit, prefetchable) [size=8M] Capabilities: <access denied> Kernel driver in use: mlx4_core Kernel modules: mlx4_core


When I plug a 10Gb Ethernet connector, I see a kernel message and then a new network device is available. However when a plug a 40Gb connector nothing happens: no kernel message, nor network device. Why?

Tried in 14.04 with same behaviour.

Any help greatly appreciated.

Hi!

What the model of your QSFP+ cable?

Could you check your HCA’s PSID?

I found the PSID with mstflint: MT_1090120019

Your adapter is infiniband HCA(Host Channel Adapter).

http://www.mellanox.com/page/firmware_table_ConnectX3IB http://www.mellanox.com/page/firmware_table_ConnectX3IB

Your cable also FDR14 56Gb Infiniband copper cable.

It seems all of your equipment is normal.

Here is another questions.

  1. How did you connect 10Gb ethernet?

  2. Do you have QSA module?

  3. What’s your switch system?

  4. Did you check your HCA’s firmware version and port configuration mode?

Yes, using QSA module: MAM1Q00A-QSA Cable Module ETH 10GbE 40Gb/s to 10Gb/s QSFP to SFP+ (https://www.mellanoxstore.com/products/mellanox-mam1q00a-qsa-cable-module-eth-10gbe-40gb-s-to-10gb-s-qsfp-to-sfp.html https://www.mellanoxstore.com/products/mellanox-mam1q00a-qsa-cable-module-eth-10gbe-40gb-s-to-10gb-s-qsfp-to-sfp.html )

And a 10G SFP+ DAC (https://www.flexoptix.net/en/universal-dac-aoc/passive-sfp-copper-cable-dac-for-10gigabit.html?300=1108&co8322=53622 https://www.flexoptix.net/en/universal-dac-aoc/passive-sfp-copper-cable-dac-for-10gigabit.html?300=1108&co8322=53622 )

No I did not check the firmware version. But this card was before in a Windows box and was working fine with the 40G.

The thing I find odd is that a few seconds after inserting the 10G (not connected at the other end) I see a kernel message and a network device becomes available. Whereas with the 40G, nothing happens at all.

Thanks for your replying.

The card Part number: MCX354A-FCBT (not sure how to find the PSID, ibstat returns nothing under 16.04, was fine under 14.04, will reboot with 14.04 next week).

The cable is a MC2207130-002 Passive Copper Cable VPI Up To 56Gb/s QSFP 2m. (https://www.mellanoxstore.com/products/mellanox-mc2207130-002-passive-copper-cable-vpi-up-to-56gb-s-qsfp-2m.html https://www.mellanoxstore.com/products/mellanox-mc2207130-002-passive-copper-cable-vpi-up-to-56gb-s-qsfp-2m.html )

Seb