How to calculate RDMA READ data transferring rate?

Infrastructure & Networking Software And Drivers
1

I want to calculate the data transferring rate of RDMA. Using ConnectX-5 whose bandwidth is declared 200Gbps. I write data transferring code with RDMA read operation. I set start timestamp just before ibv_post_send(), and set end timestamp just after I get cq whose opcode is IBV_WC_RDMA_READ. Calculating the rate, I got transferring rate at about 268.4 Tbps which surpasses the limit. I came out that getting a CQ element dose not mean the data actually mapped to local. So use memcpy copy data once and calculate time t1=end-begin; Then I use memcpy copy same size of data and calculate mere copy time t2. Rate=data_size/(t1-t2). Still, I got 185.6Gps which also surpass the limit. However, I use ofed ib_read_bw to get transferring rate of 22.122Gbps

Question: 1. Does the way I set timestamp right? If not, how to set timestamp right or how to design experiment to calculate transferring rate. 2. What does 100Gps transferring rate, which is declared by Mellanox, mean? Is packet transferring rate or read/write/send/receive operation?

​

2

Hello Gan -

You may need to examine the source code from MOFED or maybe the below links to answer the programming questions.

http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf

https://www.csm.ornl.gov/workshops/openshmem2013/documents/presentations_and_tutorials/Tutorials/Verbs%20programming%20tutorial-final.pdf

I am curious to see where you are getting the 100Gps transferring rate. Did you mean 100Gb/s? Do you have a link? Wa it a typo?

Thanks - steve

3

Hello Gan -

Please see perftest for benchmarking: https://community.mellanox.com/s/article/perftest-package

Thanks,

~Steve