I don’t see any obvious problems there, but IIRC there is a lot of config required to make this work… I guess for completeness you could show us how you’ve configured your host NICs and drivers and same for guests? And relevant flow control settings on your switch/es, host and guests (I believe you might need to be setting qos parameters in the guest if you are not already)? Are you using the same OFED inside the guests/VMs?
Following the output I would expect the same results in both cases, however A->B_vm is good, but B_vm->A is bad. Having ~3.5G in one direction shows that there is no issues with IB communiction, so may be it is how the ranks are bound on the VM or real host?
mpirun -np 2 -host A,B_vm : 3554.95 MB/s
mpirun -np 2 -host B_vm,A : 804.30 MB/s
Try to use ib_read_bw and ib_send_bw utilities before MPI. Also check that your CPU are running on the maximum speed, as seems that they are not - 1200 MHz/1995Mhz.
00:04.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
…
To check result on both 2 CentOS 6.5 hosts and 2 CentOS 7.1 hosts, I’m using same machines/setup. Even though I tried to make new VM images (CentOS 6.5/7.1) again, the bandwidth result is still bad. Should I have to do something more (that’s not needed on CentOS6.5 host) on CentOS7.1 to get reasonable bandwidth between 2VMs? Could you please explain the QoS parameters in the guest?? I tried same guest/VM (default QoS) on CentOS6.5 and CentOS7.1. Only the VM on CentOS7.1 have a problem.
Are you sure your issue is not related to other virtualisation factors? E.g., are you pinning your VMs to CPU cores and exposing the host NUMA topology to them? If your VMs have memory accesses that cross NUMA nodes (e.g., need to cross QPI) then that would explain your performance degradation as the message size increases and the effect of CPU caches is reduced to be dominated by memory.
Thanks, Blair. Surely I checked the NUMA topology. I have two NUMA nodes and each has 8 cores. While pinning VM to different cores/NUMA nodes, I’ve checked the MPI Bandwidth Performance. In case of 1B~32KB, the performance is too bad (<17% of host result) even though maximum bandwidth is OK in case of >=64KB.
What does the NUMA topology look like inside your VMs, i.e., are you pinning memory nodes as well as CPUs? Do you have cpu numa xml elements in your libvirt domain xml, e.g., like:
Your figures make it look like there might be a problem with hostB’s VM… do you get the same results (hostA<->hostB’s VM (3521.03 MB/s)) when reversing to hostA’s VM<->hostB?
It might be useful if you dump more of your config here, e.g., lscpu/numactl -h on the hosts and inside the VMs, the libvirt xml etc.