How to I/what is the easiest way to add a second Sx6012 switch to my setup? I have another Sx6012 and 3 nodes.

Infrastructure & Networking Switches and Gateways
1

Hello. How to I/what is the easiest way to add a second Sx6012 switch to my setup? I have another Sx6012 and 3 nodes.

I have 3 server nodes each having 1 dual port ConnectX-4 adapter. I’d like to use the second port on each of these adapters and connect them to a second Sx6012 to add HA to this setup.

Can someone give me the cliff notes version on what to expect with this setup? I installed the first setup, so I should be fine doing this as well. I have all components ready to go. Thanks to anyone willing to help. Thanks

2

Hello Steven,

Thank you for posting your inquiry on the NVIDIA Networking Community.

Based on the information provided, you can use the instructions provided in from the following link to setup HA between the two switches (Make sure the switches are on the same code level, and we recommended the latest MLNX-OS code available for this p/n ; MLX-OIS 3.6.8012)

Instructions on how to configure →

https://community.mellanox.com/s/article/understanding-subnet-manager--sm--high-availability--ha--on-mellanox-infiniband-switches#jive_content_id_Mellanox_SM_HA_Solution_Mellanox_InfiniBand_Switches

Later UM description of SM HA including diagram → https://docs.mellanox.com/display/MLNXOSv390606/Subnet+Manager+High+Availability

Connect the two switches together and connect the 2nd port of the HCA to switch new switch, and you are good to go. (See diagram below)

927594For creating the LAG on the adapter, please follow the instructions from the following link → https://community.mellanox.com/s/article/howto-create-linux-bond--lag--interface-over-infiniband-network

When using this setup with IPoIB, you will have an active/standby configuration. If using RDMA traffic, you can run traffic on both ports of the adapter.

Thank you and regards,

~NVIDIA Networking Technical Support