I picked up a pair of ConnectX VPI cards along with CX4->SPF+ cable as a cost-effective way to get 10gigE for shared storage.
I initially plugged them into an switch (gigE switch with 4 10gigE uplink ports). To eliminate switch or the CX4->SFP+ cable as the cause of the problem, I took the two systems with same ConnectX cards and connected the two cards directly with a passive CX4 cable.
With this direct card-to-card connection and tested via iperf I get these results.
IPoIB: Around 6-7Gbit/s with datagram mode and default 2044 MTU. Switching to connected mode and setting MTU to 65520 doesn’t make any difference.
10gig Ethernet: I’m lucky to get 200-300 Kbit/sec.
Card firmware is latest available for these cards v2.9.1000.
Both systems are Debian Wheezy 7.8 (well, Proxmox)
Adapters are identified as:
04:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)
mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)
Possible causes? Solutions?