At first, we installed correct infiniband switch, infiniband cable. And we also updated our firmware as well as driver verion. But we just got 6Gps bandwidth.
After several days searching,we finally solved this problem because of our inappropriate PCI-E slot. We have already reached the rational speed via using Mellanox ConnectX3 Adapter over servers in Win10 PCs. However, we just locate the error, but we don’t know the reason why we should change the slot from PCI_E_Gen3_x16 to PCI_E_Gen2x16. According to the requirements of Mellanox ConnectX3, we had better plug IB cards in PCI_E_Gen3x8 slot.
When we plug our Infiniband Card in PCI-E _x16_3 slot, our system can’t successfully recognize correct PCI-E BUS. The system shows us our IB card are using “Bus=PCI_E Gen2 SPEED=5.0Gbps, WIDTH=x2, CAPS=8.0X8 "(PCI-EX2_2 slot). After we change our slot to PCI_E_X16_2 slot, the system can correctly recognize the PCI-E attributes which it tells us we are using the "Bus=PCI_E Gen3 SPEED=8.0Gbps, WIDTH=x8, CAPS=8.0X8”(PCI-EX8_3). In fact, we are confused, because we use the PCI_E Gen2_x16 slot. Why does it show us PCI_E_Gen3_x8 adapter attributes.
Our system is Windows10. We are using the IB tools, such as the command like “nd_write_bw”, now it can reach 31.5Gps over two servers now than what in the past we just got 6.5Gps . But we still have the question that why it can’t recognize the correct slot when we plug IB cards in PCI-E_X16_3 slot. In order to use IB card, we are using PCI_E_2_X16 slot.
And another situation we should tell is our server has a spare PCI-EX16_3 slot, a spare PCI-EX16_2 slot and a spare PCI-EX2_2 slot. At first, we use PCI-EX16_3 slot for IB card, now we are using PCI-EX16_2 slot.
We enclose adapter info on the attachment. The PDF named “PCI-E_3X16_result.pdf” describe the running result and adapter info when we using the PCI-e_3x16 slot. The PDF named “PCI-E_2X16_result.pdf” describe the running result and adapter info when we using the PCI-e_2x16 slot.
Can someone tell me the reason?We are looking forward to hearing from you.
PCI-E_3X16_result.pdf (172 KB)
PCI_E_2X16_result.pdf (142 KB)