Mellanox and Infiniband limitation


Sorry for some theoretic question but it interesting for me

In Infiniband specification wrote what Infiniband network can support for up to 64,000 addressable devices.

For a long time already the tendency to increase the number of hosts in the network (including the exaflop plans).

How will the network be organized with more than 64,000 hosts ?

Only by IB routers or Mellanox have plans to expand this limitation ?

Thanks for your time

The limitation per subnet is based on the unicast LID space. LIDs are 16 bits and unicast LIDs go from 1 up to 48K. Assuming LMC is 1, that’s 1 LID per device. To go above this limit, IB routers would be needed which would allow for multiple subnets. This extends the layer 2 (LID space) with layer 3 (GIDs) which means GRH needs to be included on any packet traversing multiple IB subnets.

One more thing:

Mellanox currently offers an IB router solution.


IB Router Architecture and Functionality

HowTo Configure IB Routers

Thanks for arise answer