Mellanox and Infiniband limitation

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hello

Sorry for some theoretic question but it interesting for me

In Infiniband specification wrote what Infiniband network can support for up to 64,000 addressable devices.

For a long time already the tendency to increase the number of hosts in the network (including the exaflop plans).

How will the network be organized with more than 64,000 hosts ?

Only by IB routers or Mellanox have plans to expand this limitation ?

Thanks for your time

2

The limitation per subnet is based on the unicast LID space. LIDs are 16 bits and unicast LIDs go from 1 up to 48K. Assuming LMC is 1, that’s 1 LID per device. To go above this limit, IB routers would be needed which would allow for multiple subnets. This extends the layer 2 (LID space) with layer 3 (GIDs) which means GRH needs to be included on any packet traversing multiple IB subnets.

3

One more thing:

Mellanox currently offers an IB router solution.

See:

IB Router Architecture and Functionality https://community.mellanox.com/s/article/ib-router-architecture-and-functionality

HowTo Configure IB Routers https://community.mellanox.com/s/article/howto-configure-ib-routers

4

Thanks for arise answer