Learning about Infiniband switches - Advice Please

1

Hello,

I’m putting in (or planning on putting in) a Ceph Cluster with my Proxmox (Virtual Hypervisor) in my home network. Between my 3 Ceph servers I want to use Infiniband and saw some decent prices on ebay. I’d like to learn more about the types of switches available and I’m hoping this community can provide some tips for me.

I’m looking at using DDR (20Gbps) Nics in my servers. Are there 8 port Infiniband switches you can suggest I look for on ebay that would work? Something that isn’t too loud. I see lots of larger 24 port switches but for my small setup 8 ports would be more than enough.

I also found an Flextronics F-X430066 8-Port 4X SDR InfiniBand External Switch for a great price but I’m assuming that this switch would limit me to 10Gbps…is this a correct assumption?

Any advice you can give would be greatly appreciated.

Thank you,

2

Charles,

How is your Ceph Cluster project going? I am getting one Flextronics F-X430066 8-Port 4X SDR InfiniBand External Switch that is arriving in two days time.

Which adapter card did you decide to use with the switch that you selected? I found this thread related to this one:

[ceph-users] Is Ceph appropriate for small installations? http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-August/004004.html

I have two Mellanox adapters MHGA28-1TC Adapter Card that communicate using ibping when connected with this cable:

Mellanox MC1104130-002 MicroGiGaCN latch, 30 AWG, 3 meter Copper Cable

And I am looking at getting two Qlogic adapters: INFINIBAND PCI-E DUAL PORT HCA 7104-HCA-128LP

My goal is to create a small cluster with good RMDA (remote DMA) performance. To me that implies the adapter cards must have memory since RDMA is not a stateless offload protocol.

I’m not certain that on-board memory in the network is a requirement for RDMA. My memory requirement limits my choice of adapter cards.

Regards,

Vincent Randal

Longmont, Colorado

3

This one is very well priced and - depending on your location - the postage is decent too:

Nortel Networks Nortel BayStack 470 (AL2012A37) 24-Ports Rack-Mountable Switch Managed stackable for sale online | eBay Nortel Networks Nortel BayStack 470 (AL2012A37) 24-Ports Rack-Mountable Switch Managed stackable for sale online | eBay

Note - I don’t have any relationship to the seller, I just typed in “Voltaire ISR 9024D-M” on eBay.

4

Hmmm, see if you can get a hold of a Voltaire ISR 9024D-M. They’re probably the best 20Gb/s speed combination of capabilities & noise. Anything that’s SDR is likely to be extremely noisy, so you’ll want to avoid them.

That being said, I haven’t used any of the Flextronics switches in person. So that one may be an exception. There’s more info about IB switches suitable for home use here, if it helps:Suggestions for quiet Infiniband switch? Infrastructure & Networking - NVIDIA Developer Forums