I have a Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6] in servers, and running RHEL 7.9 x86-64. In RHEL in the settings - network gui it offers the choice for Transport Protocol of either Datagram or Connected. It seems only Datagram mode works, and Connected mode is not supported.
Is Mellanox moving away from Connected Mode? Redhat says they do not support it in RHEL 8 and that it “does not scale”.
I have just 3 servers on a 100gbps infiniband network (this is not a 10,0000 node cluster) and I want to use connected mode. I am under the impression that connected mode should be better than datagram in my case ?
Thank you for posting your inquiry on the NVIDIA Developer Forum - Infrastructure and Networking - Section.
Yes, the information provided by RH is correct. In case of using RH INBOX driver, they currently only support datagram mode.
A few driver versions ago, we introduced Enhanced IPoIB, which will provide more better scalability and performance improvement than using CONNECTED_MODE.
Snippet from this section: " Enhanced IPoIB feature enables offloading ULP basic capabilities to a lower vendor specific driver, in order to optimize IPoIB data path. This will allow IPoIB to support multiple stateless offloads, such as RSS/TSS, and better utilize the features supported, enabling IPoIB datagram to reach peak performance in both bandwidth and latency.
Enhanced IPoIB supports/performs the following:
** Stateless offloads (RSS, TSS)*
** Multi queues*
** Interrupt moderation*
** Multi partitions optimizations*
** Sharing send/receive Work Queues*
** Vendor specific optimizations*
** UD mode only"*
Still it is based on your use case if CONNECTED_MODE will give you more performance increase. The performance increase you will see is only for ULP, legacy IP applications.