I am new to InfinBand and working on my final year project, in which at initial, I have to configure IPOIB and RDMA over InfiniBand. I have configured both of them in Oracle Linux and status of RDMA is “active” and it is “enabled” (see capture 1 and capture 2).
I tried to test the bandwidth and run the command “ib_write_bw --reprt_gbits --iters=100000 --bidirectional”, and waited for client to connect. The result of average bandwidth came out to be around 20 Gbps (see capture 3).
Firstly, I want to know, whether my RDMA configuration is working fine or do I need to run other scripts or commands as well to verify RDMA ?
Secondly, if it is working fine, but I am getting average bandwidth of 20 Gbps. However, in actual InfiniBand provides around 40 Gbps link. I have also increased the number of iterations, but it didn’t effect the average bandwidth.
I am utilizing one port of IB card on each system. So, is that reason of getting 20 Gbps bandwidth or does each of two port supports 40 Gbps link?
In our lab, we tested with the MLNX_OFED installed as that is what we support. It comes with all the latest ‘perftest’ utils.
Attached is a screenshot which I took from our lab system, it show ‘ib_write_bw’ bi-directional for a 50Gb/s link. And as it is bidirectional it will add both results together.
