Connectx-3 VPI Nics don't work properly on ASUS Z8PE-D12X motherboards

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

Hi,

there is a small compuring cluster, consisting of 5 homogeneous nodes, which include: mb Z8pe-d12x, 2 CPU Xeon X5680, RAM 96gb (12x8), raid controller Pike 2008, drives 3 ssd, 3 sata, video nvidia quadro k5000, network card Connectx-3 (MCX354A-FCBT) infiniband (switch MIS5022, ib optical cables MC2207310). OS - windows 7 x64 pro). The problem is that the ngrateful etwork cards run at a reduced speed of 2.5GT/s instead of the expected 5GT/s (PCI-E 2.0). Accordingly, Infiniband speed ~ 14Gbit/s (instead of ~ 30Gbit/s). The latest firmware and drivers are installed (MB bios - 1401, network firmware - 2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin, network driver - MLNX_VPI_WinOFl_v5.35. Other devices are detected properly. I tried to remove other devices and install a network card into various pci-e slots - the result didn’t change

#lspci:

LnkCup: Speed 8GT/s, Width x8

LnkSta: Speed 2.5GT/s, Width x8#

Are there any thoughts on solving this problem? Or the reason is incompatibility?

I would be grateful for any help!

2

Hi,

The LnkCap show the potential capabilities of your link, the LnkStat show the current status of your link.

The lnk status reflects on the speed of your current PCI slot, try to move your card to another slot.

To identify your issue, use lspci -tv and lspci -s -xxxvvv

You will see where your card is regarding his root complex PCIe.

With the second command, search for Gen1, and you will see that your card is identified as a Gen1 PCIe.

Marc

3

I tried to remove other devices and install a network card into various pci-e slots - the result didn’t change.

I got a respnse from ASUS: Re: About MB Z8PE-D12X with Mellanox Connectx-3 VPI Nics

Hello!

unfortinately, Z8PE-D12X is a very old MB model and out of support.

probably you are right about uncompatibility: MCX354A-FCBT is PCI-E gen 3 cards, but Z8PE-D12X supports gen 2 as max speed.

During PCI-E negotiation, system define maximum speed as PCI-E gen 1.