Ubuntu server 16.04.02 fresh install on a SuperMicro 7048GR-TR
ConnectX-3 dual port board.
It is recognised fine:
lspci -v | grep -i mellan -A 10 82:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3] Physical Slot: 21 Flags: bus master, fast devsel, latency 0, IRQ 35 Memory at fb100000 (64-bit, non-prefetchable) [size=1M] Memory at 387ff2800000 (64-bit, prefetchable) [size=8M] Capabilities: <access denied> Kernel driver in use: mlx4_core Kernel modules: mlx4_core
When I plug a 10Gb Ethernet connector, I see a kernel message and then a new network device is available. However when a plug a 40Gb connector nothing happens: no kernel message, nor network device. Why?
Tried in 14.04 with same behaviour.
Any help greatly appreciated.
Hi!
What the model of your QSFP+ cable?
Could you check your HCA’s PSID?
I found the PSID with mstflint: MT_1090120019
Your adapter is infiniband HCA(Host Channel Adapter).
Firmware for ConnectX®-3 IB/VPI Firmware for ConnectX®-3 IB/VPI
Your cable also FDR14 56Gb Infiniband copper cable.
It seems all of your equipment is normal.
Here is another questions.
-
How did you connect 10Gb ethernet?
-
Do you have QSA module?
-
What’s your switch system?
-
Did you check your HCA’s firmware version and port configuration mode?
Yes, using QSA module: MAM1Q00A-QSA Cable Module ETH 10GbE 40Gb/s to 10Gb/s QSFP to SFP+ (https://www.mellanoxstore.com/products/mellanox-mam1q00a-qsa-cable-module-eth-10gbe-40gb-s-to-10gb-s-qsfp-to-sfp.html https://www.mellanoxstore.com/products/mellanox-mam1q00a-qsa-cable-module-eth-10gbe-40gb-s-to-10gb-s-qsfp-to-sfp.html )
And a 10G SFP+ DAC (10G SFP+ DAC | Passive Copper Cable, 0.5 - 10 m - Universal AOC & DAC 10G SFP+ DAC | Passive Copper Cable, 0.5 - 10 m - Universal AOC & DAC )
No I did not check the firmware version. But this card was before in a Windows box and was working fine with the 40G.
The thing I find odd is that a few seconds after inserting the 10G (not connected at the other end) I see a kernel message and a network device becomes available. Whereas with the 40G, nothing happens at all.
Thanks for your replying.
The card Part number: MCX354A-FCBT (not sure how to find the PSID, ibstat returns nothing under 16.04, was fine under 14.04, will reboot with 14.04 next week).
The cable is a MC2207130-002 Passive Copper Cable VPI Up To 56Gb/s QSFP 2m. (https://www.mellanoxstore.com/products/mellanox-mc2207130-002-passive-copper-cable-vpi-up-to-56gb-s-qsfp-2m.html https://www.mellanoxstore.com/products/mellanox-mc2207130-002-passive-copper-cable-vpi-up-to-56gb-s-qsfp-2m.html )
Seb