Currently, we are using ConnectX-5 NICs slotted in gen 3 PCIe x16 slots. They are dual ported, although we are attempting to utilize only 1 of the channels to use all 16 lanes available in the PCIe.
The expected network bandwidth that we should be getting is ~80 Gb/s to 100 Gb/s. Instead we are reaching ~60 Gb/s or less. This is due to the PCIe still thinking we have 2 ports plugged in rather than only 1, so we are only utilizing 8 lanes instead of 16.
This is referenced in the article: PCIe configuration for max performance, where the max PCIe bandwidth is calculated to be
Maximum PCIe Bandwidth = 8G * 8 * (1 - 2/130) - 1G = 64G * 0.985 - 1G = ~62Gb/s.
Which explains why we are only getting ~60Gb/s. Again, the goal is to try to utilize all 16 lanes to increase the bandwidth.
For reference we also have a single port ConnectX-5 that is also connected to a PCIe x16 on a 6 core Intel(r) Xeon(R) W-2133 CPU @ 3.60Ghz w/ 64 GB of RAM. This ConnectX-5 produces ~80Gb/s.
The dual port ConnectX-5 are on similar machines with the same specs.
I already sent in a case ticket, but we figured that making a thread would also help.