I have a ceph cluster with 5 servers connected to my MQM8700 IB switch. using connect5-Ex cards on the servers,
I’m trying to get them to work by creating a network bridge on the ibX interfaces but the network bridge doesn’t work.
after creating the network bridges using netplan, the bridge does not get a static ip address as expected.
I’m trying to deploy the ceph cluster with openstack so the br-storage bridge is needed.
Any help will be appeciated !!
Hi jeffaw99,
Unfortunately, Infiniband to Ethernet bridges are not supported in this fashion.
We do have a product available for this use case called the Skyway - this is an IB to Ethernet gateway appliance:
If you have questions, please reach out to our Sales and Solutions teams - they will be able to assist you with developing a solution to fit your unique business needs:
Thanks, and best regards,
NVIDIA Enterprise Experience
I’ve had success using OVS and creating the patching between OVS switches across an IPoIB network. You could then create patches to your Ethernet NICs from the same switches if you need to cross that boundary into the native ethernet networks…
If you just want an IP network obviously IPoIB alone would suffice. If you’re looking for a way to deploy systems onto hardware using something that checks for 48-bit mac addresses like MaaS then this won’t work. The next step would be eIPoIB but I have no experience with this. Any of these configurations would require custom commissioning scripts too.