Why do larger bytes lead to smaller bandwidth in ib_read_bw testing?

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi all:

I have two servers, each with a CX7 NIC. These two servers are interconnected through the RoCE network.

I run ib_read_bw test, the output is:

bytes iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 1000 0.048487 0.047362 2.960138
4 1000 0.082478 0.081424 2.544504
8 1000 0.16 0.16 2.488794
16 1000 0.31 0.31 2.459401
32 1000 0.66 0.66 2.577914
64 1000 1.30 1.30 2.540126
128 1000 2.64 2.64 2.577497
256 1000 5.29 5.29 2.581087
512 1000 10.40 10.39 2.537566
1024 1000 20.98 20.95 2.557972
2048 1000 36.49 36.47 2.225960
4096 1000 44.37 44.37 1.353933
8192 1000 77.33 77.21 1.178198
16384 1000 108.69 108.65 0.828914
32768 1000 139.85 139.85 0.533492
65536 1000 155.72 155.37 0.296340
131072 1000 168.72 168.66 0.160848
262144 1000 185.75 185.75 0.088572
524288 1000 262.96 172.41 0.041107
1048576 1000 182.38 182.36 0.021739
2097152 1000 190.56 190.55 0.011358
4194304 1000 201.90 201.90 0.006017
8388608 1000 222.14 222.14 0.003310

The BW average is 185.75Gbs when bytes=262144，
And the BW average is 172.41Gbs when bytes=262144，

What confuses me is why larger bytes result in smaller bandwidth.

I performed the ib_write_bw test, and the results of this test does not show this phenomenon.

Can anyone provide a reasonable explanation?

2

Hello @Chely_John,

Thank you for posting your query on our community. The “Read Requestor Lock” is a mechanism used in RDMA operations to manage and synchronize the access to the QP hardware context during RDMA Read operations.

RDMA Write operations are generally more efficient than RDMA Read operations because they do not involve the same level of synchronization and acknowledgment mechanisms.

RDMA Write involves sending data packets and receiving acknowledgments.

RDMA Read involves receiving data packets, locking and frequently updating the QP context, and then unlocking it. Also, if the buffer size is larger than the MTU, smaller MTU sizes result in more packets and thus more frequent updates to the QP hardware context.

Hope this helps.

Thanks,
Bhargavi