I’m currently considering upgrading the network of our group clusters, which contains several NUMA nodes (2 sockets) with single IB card. And I wonder if it can benefit from multi infiniband cards?
Say our server uses PCIe3.0 which limits the max bandwidth to 128GB/s. But if we install two EDR 100GB IB cards (1 for each socket), will the network bandwidth that across the nodes be expanded to 200GB/s ? Will the process communications enforced go through their local IB and use the high speed interconnection within the same NUMA node?
I have not found any documentation related to this topic, does anyone have technical supports or some successful cases?
Thank you for posting your query on our commnuity. I am not sure if you had a chance to refer to the following article which explains the mapping between PCIe, port, NUMA node and the driver.
Sorry for the late reply! I find this web page useful for checking the ib card connection and constraining processes on specific NUMA node.
This is great but I am wondering how this could be applied to a cluster? Say would the UCX library automatically use the local ib when running MPI on dual ib nodes? If not, what parameter is needed? Are there any tools to monitor or check if processes are using the right ib card? Since it is hard to use taskset to running across nodes via batch system like SLURM.