Fat-tree based IBA architecture

If you are sending packets to the internet through a switch gateway, how do the packets from several processing nodes simultaneously be routed through the gateway since the gateway has only one Target Channel Adapter (TCA)


See this VPI Gateway Considerations https://community.mellanox.com/s/article/vpi-gateway-considerations post, it explains it very nicely.

The Gateway has one TCA, this is an internal block within the switch.

The gateway can switch IB to Eth traffic (from any number of nodes), same as regular Ethernet or IB.

The impact of fact that there is only one TCA is related to routing on the infiniBand side and the utilization of the links connected to the gateway (depends on your topology)

See for example consideration 2 in VPI Gateway Considerations https://community.mellanox.com/s/article/vpi-gateway-considerations .

let me know if you have more questions,

Could you share your network/setup so we can discuss more into specific details?



How then can one connect to cloud/internet? I thought gateways are used for connection between a particular network to the internet.

Are there send-receive queue pairs (QPs) at the IBA router? I am really confused about how packets are routed from IBA subnet to the cloud.

I have gone through the VPI Gateway Considerations. https://community.mellanox.com/s/article/vpi-gateway-considerations

But, if I were to send packets from this topology to/across the internet, will I use a gateway or a router?

Thanks very very much for your help.

The word Gateway in general used in many cases across the industry. in this case we use it when connecting IB networks to Ethernet networks. The gateway is not IP router (nor IB router) and have QPs needed to the IB side.

Assuming your servers send IPoIB traffic, You need a gateway to transfer IPoIB traffic to IPoEth and a router (default gateway) to terminate the IP packets and pass it to a different network (e.g. in the Enterprise, probably you will have some kind of firewall or NAT services before going into the ISP.

See also this document - see the ARP flow.

http://www.mellanox.com/related-docs/prod_gateway_systems/Configuring_Mellanox_Hardware%20_for_VPI_Operation_Application… http://www.mellanox.com/related-docs/prod_gateway_systems/Configuring_Mellanox_Hardware%20_for_VPI_Operation_Application_Note.pdf



Thanks. I quite understood what you’ve said, and after going through the article you sent me - it was helpful. However, do you think that the Router (default Gateway) could possibly have TCA and send and receive queue pairs?

I want to know what destination Local identifier (DLID) the packets from the processing nodes (source) have so that the switches can forward packets correctly to the Gateway before it finds its way across the internet using Global Identifier (GID) to the destination.



Here is some more information and maybe this will simplify and answer your question:

Mellanox Gateway product is basically a proxy-arp device (as far as IP goes); means - it will route packets in and out of the InfiniBand network based on source and destination ARP - packet by packet.

it has no awareness at session level therefore, it is not aware of streams. It can only forward back and forth IP packets (of any kind) which means that RDMA will necessarily stay in the IB fabric.

hope this helps…