What's the deal with infiniband CONNECTED Mode?

I have a Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6] in servers, and running RHEL 7.9 x86-64. In RHEL in the settings - network gui it offers the choice for Transport Protocol of either Datagram or Connected. It seems only Datagram mode works, and Connected mode is not supported.

Is Mellanox moving away from Connected Mode? Redhat says they do not support it in RHEL 8 and that it “does not scale”.

I have just 3 servers on a 100gbps infiniband network (this is not a 10,0000 node cluster) and I want to use connected mode. I am under the impression that connected mode should be better than datagram in my case ?

Hello ron7000cg,

Thank you for posting your inquiry on the NVIDIA Developer Forum - Infrastructure and Networking - Section.

Yes, the information provided by RH is correct. In case of using RH INBOX driver, they currently only support datagram mode.

A few driver versions ago, we introduced Enhanced IPoIB, which will provide more better scalability and performance improvement than using CONNECTED_MODE.

You can review the following direct link to the driver UM → IP over InfiniBand (IPoIB) - MLNX_OFED v5.5- - NVIDIA Networking Docs
to better understand the benefits when using Enhanced IPoIB mode (datagram).

Snippet from this section:
" Enhanced IPoIB feature enables offloading ULP basic capabilities to a lower vendor specific driver, in order to optimize IPoIB data path. This will allow IPoIB to support multiple stateless offloads, such as RSS/TSS, and better utilize the features supported, enabling IPoIB datagram to reach peak performance in both bandwidth and latency.

Enhanced IPoIB supports/performs the following:

** Stateless offloads (RSS, TSS)*
** Multi queues*
** Interrupt moderation*
** Multi partitions optimizations*
** Sharing send/receive Work Queues*
** Vendor specific optimizations*
** UD mode only"*

Still it is based on your use case if CONNECTED_MODE will give you more performance increase. The performance increase you will see is only for ULP, legacy IP applications.

The MLNX_OFED driver still supports CONNECTED_MODE and you can definitely run your setup with this setting → IP over InfiniBand (IPoIB) - MLNX_OFED v5.5- - NVIDIA Networking Docs

Thank you and regards,
~NVIDIA Networking Technical Support

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.