Does OpenSM supports switchless torus based IB networks ? reference range of time required for different phases of openSM

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hello All,
I was wondering if OpenSM can discover the SWITCHLESS torus topology (i.e. directly connected multiport HCAs in torus fashion without switches).

Also is there a lead/doc/literature about timing performance analysis of different phases of OpenSM like subnet discovery, LID assignments, switch configuration etc.

2

Hello maheshworkaddr,

Thank you for posting your inquiry on the NVIDIA Developer Forum - Infrastructure and Networking - Section.

In the IB technology switchless Torus topology does not exist.

InfiniBand uses a switched fabric topology, which is different from the older shared medium Ethernet. This means all transmissions go through a switch, making it more efficient and scalable.

There are a few common InfiniBand topologies:

  • Fat Tree: This is a very popular choice for high-performance computing (HPC) clusters. It’s designed to provide high bandwidth and low latency, but it can be expensive to implement.
  • Mesh: In a mesh topology, each switch is connected to every other switch. This provides high redundancy and fault tolerance, but it can also be complex and expensive.
  • Torus: This topology is often used in larger clusters. It offers good scalability and performance, but it can be more difficult to manage than other topologies.
  • Ring: A ring topology connects each switch to two others, forming a ring. It’s a simple and cost-effective option, but it’s not as scalable as other topologies.

The choice of topology depends on the specific needs of the cluster, such as the number of nodes, the required bandwidth, and the budget.

In addition to these basic topologies, there are also hybrid topologies that combine elements of different topologies. This allows for greater flexibility and customization.

Here are some resources where you can learn more about InfiniBand topologies:

It seems you’re interested in understanding how the performance of a Subnet Manager (SM) in an InfiniBand network can vary across its different operational stages. This is a great question, as SM performance is crucial for overall network efficiency, especially in demanding environments like HPC clusters.

Here’s a breakdown of how SM performance can be impacted during key stages:

1. Initialization/Startup:

  • Discovery and Initialization: This stage involves the SM discovering all the devices (switches, HCAs) in the fabric and assigning them unique identifiers (LIDs). A large fabric can prolong this stage, impacting how quickly the network becomes fully operational. Optimized SM implementations use efficient discovery protocols and parallel processing to speed up this process.
  • Routing Table Calculation: The SM calculates the optimal paths between all pairs of devices. The complexity and time taken for this depend on the fabric size and topology. Efficient routing algorithms and sufficient processing power are essential for quick startup, especially in dynamic environments where the topology might change.

2. Steady State Operation:

  • Event Handling: The SM continuously monitors the fabric for events like device addition/removal, link failures, and congestion. Efficient event handling ensures quick response and minimal disruption to ongoing communication.
  • Query Processing: Applications and management tools frequently query the SM for information about the fabric. The SM’s ability to handle these queries efficiently without impacting core functions is crucial.
  • Maintaining Fabric State: The SM needs to maintain an up-to-date view of the fabric state. This involves processing link state updates, managing routing tables, and ensuring consistency.

3. Under Stress/Failure Scenarios:

  • Failure Recovery: When a link or device fails, the SM needs to quickly recalculate routes and reconfigure the fabric to maintain connectivity. Fast failover mechanisms and efficient recovery algorithms are critical for minimizing downtime.
  • Congestion Management: Under heavy load, the SM might need to implement congestion control mechanisms to prevent performance degradation. The effectiveness of these mechanisms and their impact on overall performance are important considerations.

Factors Influencing SM Performance:

  • SM Implementation: The specific SM software (e.g., OpenSM, Mellanox SM) and its configuration significantly impact performance.
  • Hardware Resources: The CPU, memory, and I/O capabilities of the machine hosting the SM play a vital role.
  • Fabric Size and Topology: Larger and more complex topologies increase the processing load on the SM.
  • Traffic Patterns: The volume and nature of traffic flowing through the fabric can affect SM performance.

To learn more about specific SM implementations and their performance characteristics, you can refer to the documentation provided by Nvidia and OpenFabrics Alliance. You can also find research papers and articles discussing SM performance optimization techniques.

I hope this information is helpful!

Thank you and regards,
~NVIDIA Networking Technical Support

3

Thanks MvB for the elaborate reply.

what i deduced from your reply is: OpenSM may not be able to discover the subnet, if deployed in cluster which is connected by IB supporting Interconnect but having switchless torus topology.

Other part of question was about readings/figures for OpenSM performance for various stages. Anyway, thanks for your suggestion to find research papers and articles about same.