I am new to InfinBand and working on my final year project, in which at initial, I have to configure IPOIB and RDMA over InfiniBand. I have configured both of them in Oracle Linux and status of RDMA is “active” and it is “enabled” (see capture 1 and capture 2).
I tried to test the bandwidth and run the command “ib_write_bw --reprt_gbits --iters=100000 --bidirectional”, and waited for client to connect. The result of average bandwidth came out to be around 20 Gbps (see capture 3).
Firstly, I want to know, whether my RDMA configuration is working fine or do I need to run other scripts or commands as well to verify RDMA ?
Secondly, if it is working fine, but I am getting average bandwidth of 20 Gbps. However, in actual InfiniBand provides around 40 Gbps link. I have also increased the number of iterations, but it didn’t effect the average bandwidth.
I am utilizing one port of IB card on each system. So, is that reason of getting 20 Gbps bandwidth or does each of two port supports 40 Gbps link ?
Thank you for posting your question on the Mellanox Community.
Based on the information provided, it shows that the RDMA connection is working else, you will not get any output.
As you are running, Oracle Linux provided MLNX drivers, we recommend to install the MLNX_OFED driver for OL, which you can find through the following link → http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers .
In our lab, we tested with the MLNX_OFED installed as that is what we support. It comes with all the latest ‘perftest’ utils.
Attached is a screenshot which I took from our lab system, it show ‘ib_write_bw’ bi-directional for a 50Gb/s link. And as it is bidirectional it will add both results together.
If after installation of our MLNX_OFED driver, you still experiencing issues, please open a Mellanox Support Ticket, by sending an email to email@example.com mailto:firstname.lastname@example.org
Thanks and regards,
~Mellanox Technical Support