I am new to InfinBand and working on my final year project, in which at initial, I have to configure IPOIB and RDMA over InfiniBand. I have configured both of them in Oracle Linux and status of RDMA is “active” and it is “enabled” (see capture 1 and capture 2).
I tried to test the bandwidth and run the command “ib_write_bw --reprt_gbits --iters=100000 --bidirectional”, and waited for client to connect. The result of average bandwidth came out to be around 20 Gbps (see capture 3).
Firstly, I want to know, whether my RDMA configuration is working fine or do I need to run other scripts or commands as well to verify RDMA ?
Secondly, if it is working fine, but I am getting average bandwidth of 20 Gbps. However, in actual InfiniBand provides around 40 Gbps link. I have also increased the number of iterations, but it didn’t effect the average bandwidth.
I am utilizing one port of IB card on each system. So, is that reason of getting 20 Gbps bandwidth or does each of two port supports 40 Gbps link ?
In our lab, we tested with the MLNX_OFED installed as that is what we support. It comes with all the latest ‘perftest’ utils.
Attached is a screenshot which I took from our lab system, it show ‘ib_write_bw’ bi-directional for a 50Gb/s link. And as it is bidirectional it will add both results together.