Can't get full FDR bandwidth with Connect-IB card in PCI 2.0 x16 slot

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
,
1

Greetings,

We have some nodes (Dell R415s running RHEL 6.8) with Connect-IB cards in a PCI 2.0 x16 slot (the only one available) and can’t seem to get more than 45 Gbit/s using ib_send_bw. I have two of the nodes connected directly using a new FDR cable and with the SM running on one of the nodes. I have updated the BIOS, the OFED and the HCA firmware on both nodes. Still I can’t seem to get the full FDR bandwidth. The connect-ib product page (http://www.mellanox.com/page/products_dyn?product_family=142&mtag=connect_ib http://www.mellanox.com/page/products_dyn?product_family=142&mtag=connect_ib ) states the following:

“Connect-IB also enables PCI Express 2.0 x16 systems to take full advantage of FDR, delivering at least twice the bandwidth of existing PCIe 2.0 solutions.”

Since the PCIe 2.0 x16 is able to support 64 Gbit/s in one direction, shouldn’t I be able to achieve full FDR (~54ish Gbit/s) as the product page implies. Or, am I wrong and there is some extra overhead that reduces the bandwidth for PCIe 2.0 x16 vs. PCIe 3.0 x16?

I have gone through the Performance Tuning for Mellanox Adapters https://community.mellanox.com/s/article/performance-tuning-for-mellanox-adapters guide and there isn’t much more that I can try based on this. The latest BIOS has nowhere near the number of setting that are suggested to be tweaked in the guide. I have also tried mlnx_tune and get one warning:

Connect-IB Device Status on PCI 01:00.0

FW version 10.16.1200

OK: PCI Width x16

Warning: PCI Speed 5GT/s >>> PCI width status is below PCI capabilities. Check PCI configuration in BIOS. <--------------

PCI Max Payload Size 128

PCI Max Read Request 512

Local CPUs list [0, 1, 2, 3, 4, 5]

But this is probably correct since I am using a PCIe 2.0 x16 slot (PCIe 2.0 can only do 5 GT/s), right?

Here is the output of ibv_devinfo:

hca_id: mlx5_0

transport: InfiniBand (0)

fw_ver: 10.16.1200

node_guid: f452:1403:002e:eb40

sys_image_guid: f452:1403:002e:eb40

vendor_id: 0x02c9

vendor_part_id: 4113

hw_ver: 0x0

board_id: MT_1220110019

phys_port_cnt: 1

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 1

port_lid: 2

port_lmc: 0x00

link_layer: InfiniBand

and iblinkinfo:

CA: A HCA-1:

0xf4521403002ee9f0 1 1 ==( 4X 14.0625 Gbps Active/ LinkUp)==> 2 1 “B” ( )

CA: B HCA-1:

0xf4521403002eeb40 2 1 ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 1 “A” ( )

Can anyone tell me if this is the best I can expect or is there something else I can change to achieve FDR bandwidth with these HCAs?

Thanks in advance!

Eric

2

Server:

ib_send_bw -a -F --report_gbits

Client:

ib_send_bw -a -F --report_gbits

Please let me know your results and thank you…

~Steve

3

Hi Steve,

Here you are. Thanks for taking an interest. Still getting ~45 Gb/sec on both client and server. Here is the client output:

[root@vx01 ~]# ib_send_bw -a -F --report_gbits vx02

Send BW Test

Dual-port : OFF Device : mlx5_0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

TX depth : 128

CQ Moderation : 100

Mtu : 4096[B]

Link type : IB

Max inline data : 0[B]

rdma_cm QPs : OFF

Data ex. method : Ethernet

local address: LID 0x3e4 QPN 0x005e PSN 0x239861

remote address: LID 0x3e6 QPN 0x004c PSN 0xed4513

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

2 1000 0.098220 0.091887 5.742968

4 1000 0.20 0.19 6.037776

8 1000 0.40 0.39 6.071169

16 1000 0.78 0.67 5.220818

32 1000 1.53 1.43 5.576730

64 1000 3.16 3.10 6.053410

128 1000 6.20 6.16 6.012284

256 1000 12.35 12.28 5.997002

512 1000 22.67 22.47 5.486812

1024 1000 38.02 36.69 4.478158

2048 1000 42.26 42.04 2.565771

4096 1000 43.82 43.68 1.332978

8192 1000 44.63 44.63 0.681005

16384 1000 44.79 44.79 0.341728

32768 1000 45.21 45.21 0.172449

65536 1000 45.35 45.35 0.086506

131072 1000 45.45 45.45 0.043342

262144 1000 45.45 45.45 0.021670

524288 1000 45.47 45.47 0.010840

1048576 1000 45.47 45.47 0.005421

2097152 1000 45.48 45.48 0.002711

4194304 1000 45.48 45.48 0.001355

8388608 1000 45.48 45.48 0.000678

Here is the server output:

[root@vx02 ~]# ib_send_bw -a -F --report_gbits

  • Waiting for client to connect… *

Send BW Test

Dual-port : OFF Device : mlx5_0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

RX depth : 512

CQ Moderation : 100

Mtu : 4096[B]

Link type : IB

Max inline data : 0[B]

rdma_cm QPs : OFF

Data ex. method : Ethernet

local address: LID 0x3e6 QPN 0x004c PSN 0xed4513

remote address: LID 0x3e4 QPN 0x005e PSN 0x239861

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

2 1000 0.000000 0.099141 6.196311

4 1000 0.00 0.20 6.229974

8 1000 0.00 0.40 6.265230

16 1000 0.00 0.69 5.362016

32 1000 0.00 1.47 5.727960

64 1000 0.00 3.22 6.283794

128 1000 0.00 6.34 6.191118

256 1000 0.00 12.64 6.169975

512 1000 0.00 23.08 5.634221

1024 1000 0.00 37.53 4.581582

2048 1000 0.00 42.63 2.602155

4096 1000 0.00 44.07 1.344970

8192 1000 0.00 45.04 0.687191

16384 1000 0.00 45.04 0.343602

32768 1000 0.00 45.35 0.172994

65536 1000 0.00 45.45 0.086690

131072 1000 0.00 45.52 0.043409

262144 1000 0.00 45.51 0.021699

524288 1000 0.00 45.52 0.010852

1048576 1000 0.00 45.52 0.005427

2097152 1000 0.00 45.53 0.002714

4194304 1000 0.00 45.53 0.001357

8388608 1000 0.00 45.53 0.000678

Please let me know if you want any other info and I will send it straight away.

Regards,

Eric

4

Hello Eric -

I hope all is well…

You won’t achieve a line rate of 56G/s because the NIC is: MCB193A-FCAT MT_1220110019 and your PCEi is 2.0

And the release notes for you FW state:

Connect-IB® Host Channel Adapter, single-port QSFP, FDR 56Gb/s,PCIe3.0 x16, tall bracket, RoHS R6

So getting ~45 ~48 Gb/s is good.

Have a great day!

Steve