Extremely slow ethernet speed with ConnectX VPI card in EN mode.

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I picked up a pair of ConnectX VPI cards along with CX4->SPF+ cable as a cost-effective way to get 10gigE for shared storage.

I initially plugged them into an switch (gigE switch with 4 10gigE uplink ports). To eliminate switch or the CX4->SFP+ cable as the cause of the problem, I took the two systems with same ConnectX cards and connected the two cards directly with a passive CX4 cable.

With this direct card-to-card connection and tested via iperf I get these results.

IPoIB: Around 6-7Gbit/s with datagram mode and default 2044 MTU. Switching to connected mode and setting MTU to 65520 doesn’t make any difference.

10gig Ethernet: I’m lucky to get 200-300 Kbit/sec.

Card firmware is latest available for these cards v2.9.1000.

Both systems are Debian Wheezy 7.8 (well, Proxmox)

Adapters are identified as:

04:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)

Module versions:

mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)

mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)

Possible causes? Solutions?

2

Hi,

I could never get native 10gigE mode to work correctly on the ConnectX-1 cards.

I guess mlx4_en driver is broken or 1st gen ConnectX support was removed as these were EOL-ed long time ago.

The two ConnectX-1 cards worked OK in IPoIB mode but that had a number of issues for me:

  • Can’t connect to a 10gigE switch (the whole point of getting these cards)

  • Couldn’t show iface bandwidth stats

  • Have to run OpenSM on one of the nodes

So I gave up on that idea.

3

Did you get these fully working? I’m actually trying to get them to work right with Proxmox 4 (Debian 8.2).