I have a MCX354A-FCBT installed on a PCI-3.0 slot of a SM X9DRi-LN4F . Ho do I make sure that FDR infinband data rate full speed on reads and writes.

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I have a MCX354A-FCBT installed on a PCI-3.0 slot of a SM X9DRi-LN4F+. How do I make sure that FDR infinband data rate full speed on reads and writes. Thanks.

2

Hi Irfan,

i am not familiar with the specific model of the server you mentioned but generally speaking this is what you need to get done for gaining the most out of your card:

  • install the card on a PCI-e Gen-3 capable PCI slot. choose the one that supports x8 or x16 speeds. Make sure gen-3 is enabled in BIOS (i’ve seen gen-3 systems that were shipped with factory defaults supporting gen-2)

  • load OS (linux or Windows) with InfiniBand driver - shoot for having the latest and greatest of either the OFED community https://www.openfabrics.org/resources/ofedrelated-downloads.html or Mellanox OFED http://www.mellanox.com/page/software_overview_ib (stick to GA versions)

  • make sure your card runs the latest firmware Firmware for ConnectX®-3 IB/VPI

  • There are few protocols that runs on top of IB. some utilize the transport efficiently; some other less. if you want to see wire-speed rates, shoot for using some native IB protocol like RDMA or other protocol that natively speaks with the IB verbs layer - this is a KEY part.since you didn’t mention what you application is like, i can’t really comment further.

That’s it - start with the above and see how far you can go. let me know…

3

If latency is important to you then NUMA affinity really makes a difference. If you run your test (i.e. ib_write_lat) in the same socket where the FDR card is connected, you can achieve latencies under 1 usec. Running in another socket will be ~20% slower. This does not affect the throughput, not in a noticeable way anyways.

IRQ affinity is also very important. BIOS setting too.

There is a very nice, and relatively short, Tuning Guide published by Mellanox which I think is a must: http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf