We observed a significant RDMA Read bandwidth degradation on ConnectX-7 when using a single RC QP with multiple outstanding RDMA Read operations.
1.Test Environment
OS: Ubuntu 24.04.2 LTS
Kernel: 6.8.0-53-generic
CPU: Intel(R) Xeon(R) 6530P
NIC: NVIDIA ConnectX-7
NIC Firmware: 28.48.1000
Transport: RC over RoCE
Test tool: ib_read_bw
2.Problem Description
When running `ib_read_bw` with a single RC QP on ConnectX-7, we observed significant bandwidth degradation at large message sizes.
The message size at which the bandwidth degradation occurs changes with the number of outstanding RDMA Reads:
`-o 16`: bandwidth drops at 128 KiB
`-o 8`: bandwidth drops at 512 KiB
`-o 4`: bandwidth drops at 1 MiB
`-o 2`: bandwidth drops at 2 MiB
`-o 1`: no bandwidth drop is observed
The degradation appears to be correlated with the number of outstanding Reads and message size.
3.Test results
- QP number = 1, Outstand reads = 16 (default)
numactl -N 0 -m 0 ib_read_bw -d mlx5_1 -a -F -q 1 -x 3 --report_gbits <server IP>
Disabling dynamic polling
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : ON Using Enhanced Reorder : OFF
TX depth : 128
CQ Moderation : 100
CQE Poll Batch : 16
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0133 PSN 0x1f61d3 OUT 0x10 RKey 0x1fff00 VAddr 0x0072cd67a01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:15
remote address: LID 0000 QPN 0x8292 PSN 0x957b0b OUT 0x10 RKey 0x38de00 VAddr 0x00758338201000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:14
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.048935 0.048678 3.042362
4 1000 0.098527 0.098439 3.076233
8 1000 0.20 0.20 3.077573
16 1000 0.39 0.39 3.069244
32 1000 0.79 0.79 3.076014
64 1000 1.57 1.57 3.070200
128 1000 3.07 3.06 2.988434
256 1000 5.99 5.98 2.921824
512 1000 11.66 11.65 2.845393
1024 1000 23.15 23.05 2.813584
2048 1000 48.50 48.49 2.959874
4096 1000 78.83 78.78 2.404261
8192 1000 160.69 160.62 2.450889
16384 1000 259.88 259.66 1.981069
32768 1000 344.33 344.25 1.313201
65536 1000 356.13 356.07 0.679152
131072 1000 353.67 32.56 0.031051
262144 1000 184.35 184.35 0.087904
524288 1000 202.59 202.59 0.048301
1048576 1000 224.67 224.67 0.026783
2097152 1000 238.68 238.68 0.014226
4194304 1000 268.67 268.66 0.008007
8388608 1000 307.87 307.87 0.004588
- QP number = 1, Outstand reads = 8
numactl -N 0 -m 0 ib_read_bw -d mlx5_1 -a -F -q 1 -x 3 -o 8 --report_gbits 172.16.6.14
Disabling dynamic polling
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : ON Using Enhanced Reorder : OFF
TX depth : 128
CQ Moderation : 100
CQE Poll Batch : 16
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 8
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0137 PSN 0xceb02d OUT 0x08 RKey 0x1ffe00 VAddr 0x0070eadda01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:15
remote address: LID 0000 QPN 0x8296 PSN 0x32a00a OUT 0x08 RKey 0x251d00 VAddr 0x0076eabdc01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:14
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.026571 0.026380 1.648749
4 1000 0.053374 0.053150 1.660935
8 1000 0.11 0.11 1.649326
16 1000 0.21 0.21 1.640041
32 1000 0.42 0.42 1.623977
64 1000 0.82 0.82 1.600880
128 1000 1.59 1.59 1.548857
256 1000 3.09 3.09 1.508603
512 1000 6.05 6.04 1.474408
1024 1000 11.95 11.94 1.457992
2048 1000 23.49 23.46 1.431698
4096 1000 48.56 48.54 1.481185
8192 1000 87.74 87.56 1.336062
16384 1000 149.02 149.00 1.136807
32768 1000 257.67 257.30 0.981530
65536 1000 333.95 333.91 0.636882
131072 1000 357.09 357.07 0.340526
262144 1000 357.59 357.58 0.170508
524288 1000 171.07 71.70 0.017094
1048576 1000 184.64 184.64 0.022011
2097152 1000 210.65 210.65 0.012556
4194304 1000 234.58 234.58 0.006991
8388608 1000 267.10 267.10 0.003980
---------------------------------------------------------------------------------------
- QP number = 1, Outstand reads = 4
numactl -N 0 -m 0 ib_read_bw -d mlx5_1 -a -F -q 1 -x 3 -o 4 --report_gbits 172.16.6.14
Disabling dynamic polling
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : ON Using Enhanced Reorder : OFF
TX depth : 128
CQ Moderation : 100
CQE Poll Batch : 16
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 4
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0136 PSN 0x2e6d5e OUT 0x04 RKey 0x1ffe00 VAddr 0x007aacdce01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:15
remote address: LID 0000 QPN 0x8295 PSN 0x9429a4 OUT 0x04 RKey 0x251d00 VAddr 0x007fa905c01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:14
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.013891 0.013858 0.866123
4 1000 0.027921 0.027878 0.871186
8 1000 0.055633 0.055556 0.868065
16 1000 0.11 0.11 0.867728
32 1000 0.22 0.22 0.868882
64 1000 0.44 0.44 0.866762
128 1000 0.88 0.88 0.860848
256 1000 1.75 1.75 0.852396
512 1000 3.49 3.48 0.848970
1024 1000 6.91 6.89 0.840872
2048 1000 13.54 13.53 0.825933
4096 1000 26.65 26.65 0.813294
8192 1000 50.10 50.07 0.763962
16384 1000 89.12 89.10 0.679789
32768 1000 146.99 146.95 0.560589
65536 1000 245.70 245.68 0.468594
131072 1000 336.89 336.85 0.321242
262144 1000 357.59 357.57 0.170503
524288 1000 357.75 357.74 0.085292
1048576 1000 171.06 101.93 0.012152
2097152 1000 192.96 192.96 0.011502
4194304 1000 217.32 217.32 0.006477
8388608 1000 251.02 251.02 0.003741
---------------------------------------------------------------------------------------
- QP number = 1, Outstand reads = 2
numactl -N 0 -m 0 ib_read_bw -d mlx5_1 -a -F -q 1 -x 3 -o 2 --report_gbits 172.16.6.14
Disabling dynamic polling
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : ON Using Enhanced Reorder : OFF
TX depth : 128
CQ Moderation : 100
CQE Poll Batch : 16
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 2
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0135 PSN 0xdf5d1c OUT 0x02 RKey 0x1ffe00 VAddr 0x007bdd0e201000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:15
remote address: LID 0000 QPN 0x8294 PSN 0x67d1f OUT 0x02 RKey 0x251d00 VAddr 0x007fccfde01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:14
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.007136 0.007135 0.445944
4 1000 0.014283 0.014282 0.446308
8 1000 0.028571 0.028570 0.446413
16 1000 0.057109 0.057095 0.446057
32 1000 0.11 0.11 0.445593
64 1000 0.23 0.23 0.444430
128 1000 0.45 0.45 0.442629
256 1000 0.90 0.90 0.438920
512 1000 1.79 1.79 0.436211
1024 1000 3.56 3.56 0.434829
2048 1000 7.04 7.04 0.429406
4096 1000 13.87 13.87 0.423148
8192 1000 26.78 26.78 0.408569
16384 1000 48.83 48.83 0.372512
32768 1000 90.08 89.94 0.343111
65536 1000 161.08 161.07 0.307207
131072 1000 263.60 263.58 0.251368
262144 1000 350.77 350.77 0.167260
524288 1000 357.80 357.78 0.085303
1048576 1000 355.84 355.84 0.042419
2097152 1000 171.28 135.48 0.008075
4194304 1000 197.25 197.25 0.005879
8388608 1000 231.72 231.72 0.003453
---------------------------------------------------------------------------------------
- QP number = 1, Outstand reads = 1
numactl -N 0 -m 0 ib_read_bw -d mlx5_1 -a -F -q 1 -x 3 -o 1 --report_gbits 172.16.6.14
Disabling dynamic polling
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : ON Using Enhanced Reorder : OFF
TX depth : 128
CQ Moderation : 100
CQE Poll Batch : 16
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 1
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0134 PSN 0xb5a158 OUT 0x01 RKey 0x1ffe00 VAddr 0x007409d7c01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:15
remote address: LID 0000 QPN 0x8293 PSN 0x134fa0 OUT 0x01 RKey 0x251d00 VAddr 0x0074f656c01000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:21:00:14
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.003559 0.003558 0.222395
4 1000 0.007119 0.007119 0.222460
8 1000 0.014226 0.014225 0.222268
16 1000 0.028447 0.028442 0.222207
32 1000 0.056817 0.056809 0.221911
64 1000 0.11 0.11 0.220837
128 1000 0.22 0.22 0.215051
256 1000 0.44 0.44 0.213251
512 1000 0.87 0.87 0.211641
1024 1000 1.72 1.72 0.209660
2048 1000 3.36 3.36 0.204928
4096 1000 6.63 6.62 0.202104
8192 1000 12.59 12.58 0.192027
16384 1000 23.43 23.42 0.178718
32768 1000 44.56 44.56 0.169989
65536 1000 77.73 77.71 0.148224
131072 1000 127.76 127.62 0.121711
262144 1000 188.67 188.66 0.089962
524288 1000 246.33 246.18 0.058693
1048576 1000 289.54 289.44 0.034504
2097152 1000 319.15 319.11 0.019021
4194304 1000 336.60 336.60 0.010031
8388608 1000 345.48 345.46 0.005148
---------------------------------------------------------------------------------------
4.Questions
- Is this a known ConnectX-7 limitation or firmware issue related to single-QP RDMA Read?
- Is there any per-QP hardware resource limit associated with the total amount of outstanding RDMA Read data, Read Response buffering, credits, or requester resources?
- Is the approximately 4 MiB threshold expected behavior on ConnectX-7?
- Are there any firmware parameters or mlxconfig settings that control this behavior