Which Connect-X 5 cards works with PCIe bifurcation and SocketDirect

Greetings,

We want to use a Supermicro motherboard that supports PCIe4 with bifurcation options in the BIOS, with 2x AMD EPYC CPUs from the 7003 generation, which allows both sockets to use the same PCIe4 slot. At least in accordance to the manual “MNL-2363.pdf”, Figure 1-4, for the motherboard H12DSi-N6.

We know/believe that we should be able to use the bifurcation feature in Socket Direct, as stated here: NVIDIA Mellanox Socket Direct Adapters | NVIDIA

Through either a connection harness that splits the PCIe lanes between two cards or by bifurcating a PCIe slot for a single card.

This is nice and all, but it’s not entirely clear which cards we can buy that provide us with this feature, namely: having a single PCIe card with which we can use the PCIe x8x8 bifurcation option in the BIOS, so that each CPU socket can use half of a single Infiniband card, via SocketDirect in said card.

Furthermore, if we can use a single card, then can we also only use a single QSFP28 port to properly use Connect-X 5 100Gbps with the Socket Direct feature, or if must use two ports.

We were thinking of purchasing the MCX555A-ECAT model, which is a PCIe3 single port card, but given the confusion of which card we can use, we don’t know if we must use the MCX556A-EDAT (PCIe4 dual port) or even the MCX556M-ECAT-S25 (without using the secondary card?).

The firmware release notes seem to imply that bifurcation is supported after a firmware update since version 16.24.1000, but the message is too vague to be certain if this refers to the same feature or not.

Many thanks! Best regards,
Bruno Santos

OK, I finally figured out where the bifurcation information in the commercial blurb was coming from: Introduction - ConnectX-6 InfiniBand/VPI - NVIDIA Networking Docs

So in other words, in Connect-X 6 there is such a card, but not in Connect-X 5.

1 Like