I have used vdbench and fio to test for RDMA connections and TCP connections.
But the weird is ,RDMA performence is worse than TCP.
Hardware and Software Used:
two machines with no switch and the cards are plugged in back to back
HCA: Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
os: CentOS Linux release 7.4.1708 (Core) 3.10.0-693.el7.x86_64
OFED: MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.4-x86_64.iso
the parameter file is:
messagescan=no
hd=default,vdbench=/root/download/vdbench50407,user=root,shell=ssh
hd=hd1,system=stor01
hd=hd2,system=stor02
fsd=fsd1,anchor=/mnt/beegfs/largetest2,depth=1,width=5,files=10,size=100m,shared=yes
fwd=format,threads=16,xfersize=1m
fwd=default,xfersize=1m,fileio=random,fileselect=random,rdpct=60,threads=16
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd1,host=hd2
rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=600,interval=1
the result is:
largetest 100MRDMATCPread(MB/s)13222008write(MB/s)883.51342total(MB/s)2205.93351.5
anyone has tested beegfs for the difference between RDMA and TCP?