Good evening,
I would like to know how to connect two infiniband switches to maximize the interconnection bandwidth.
I have an 8700 and 8790 switch. I want to connect two racks each with 20 nodes at 200gb/s. I would like to connect the two switches with 10/20 inifniband passive cables to have 2000gb/s of interconnection between the switches. Is it enough to connect the cables or do I have to set something at the software level on the 8700 switch? Which mode do you recommend to obtain this type of interconnection? Stacking or other?
Thanks
Matteo
Any suggestion?
Once you have a subnet manager running and the switches have physical connections – links should be up and running.
With regards to the BW question – assuming both switches have 40 ports, to obtain a none-blocking topology you would need 1:1 ratio of down/up links on each switch. So each switch may host 20 nodes while interconnecting with 20 ports to the other switch.
Any other configuration with 20 nodes hanging off the switch will cause a blocking scenario and won’t allow full line rate between nodes.
With regards to the configuration of the managed 8700 switch – none is needed in the simplest case of just connecting the ports to the 8790 link-partner. Unless the ports were previously configured to lower speeds, the default configuration should allow the switch to linkup in the highest rate the cable (I.e. 200gbps cable would raise 200gbps link etc.)
1 Like
Hi dwaxman,
Thank you very much for the reply. The concept of blocking and connection is clear to me. How can I verify that for example if 5 nodes go at 200gb/s and connect to as many 5 nodes in the other rack, the bandwidth is actually 5x200gb/s? How do switches figure out which way to pass traffic? Is there a routing manager or something? Technically, am I sure it works without having to claim to use interconnect cables for stacking between switches?
Regards,
Matteo
any news?
Hi dwaxman,
can you help me to understand how the switch work? How can I see if the bandwitdh is 5x200Gb/s ?
You can’t just plug in 10–20 cables and magically get 2000 Gb/s 😅. On the 8700/8790 you need to do some LAG/MLAG stuff — the switch doesn’t automatically aggregate all the links. Basically you create a port channel with all the cables, same on the other switch, and traffic gets distributed across them.
Stacking is an option, but for just two switches a normal LAG is usually simpler. Make sure the cables are in the same port group so the hashing isn’t messed up, and check the firmware supports that many links in one channel.
Bottom line: cables alone won’t do it, you need to configure it.