Can BlueField-3 B3220 P-Series (900-9D3B6-00CV) connect SSDs through the PCIe auxiliary/extension interface?

Infrastructure & Networking BlueField
1

Hi,

I would like to confirm whether it is supported and electrically/mechanically feasible to connect PCIe/NVMe SSDs to the PCIe auxiliary / extension interface of a NVIDIA BlueField-3 B3220 P-Series DPU, specifically model: BlueField-3 B3220 P-Series (900-9D3B6-00CV)

According to the BlueField-3 documentation, the PCIe Auxiliary Card Kit appears to expose Downstream Port capability:

https://docs.nvidia.com/networking/display/bf3dpu/pcie-auxiliary-card-kit

On my DPU, I can successfully configure PCIe bifurcation so that the main gold-finger interface remains x16 and the auxiliary/extension side is split into four x4 links:

sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s \
    PCI_BUS00_HIERARCHY_TYPE=0 PCI_BUS00_WIDTH=5 PCI_BUS00_SPEED=4 \
    PCI_BUS10_HIERARCHY_TYPE=2 PCI_BUS10_WIDTH=3 PCI_BUS10_SPEED=4 \
    PCI_BUS12_HIERARCHY_TYPE=2 PCI_BUS12_WIDTH=3 PCI_BUS12_SPEED=4 \
    PCI_BUS14_HIERARCHY_TYPE=2 PCI_BUS14_WIDTH=3 PCI_BUS14_SPEED=4 \
    PCI_BUS16_HIERARCHY_TYPE=2 PCI_BUS16_WIDTH=3 PCI_BUS16_SPEED=4

Based on the PCIe Auxiliary Card Mechanical Drawings and Dimensions, it seems that the auxiliary connector can be converted to a PCIe x16 edge connector using the PCIe Auxiliary Card Kit.

My questions are:

  1. Is it officially supported to connect PCIe/NVMe SSDs behind the BlueField-3 auxiliary interface when it is configured as downstream ports? Is there an NVIDIA-recommended adapter, carrier board, backplane, or PCIe switch board for converting the auxiliary interface to a standard PCIe x16 slot?

  2. If the auxiliary card exposes a PCIe x16 edge connector, can a passive female-to-female PCIe adapter/riser be used to convert it into a PCIe x16 slot, so that an SSD adapter card can be inserted?

    The intended topology is:

    BlueField-3 auxiliary connector
    |
PCIe Auxiliary Card Kit
    |
PCIe x16 edge connector
    |
passive female-to-female PCIe adapter / riser
    |
PCIe x16 slot
    |
NVMe SSD adapter card / M.2 adapter / PCIe switch card
    |
NVMe SSDs

  3. If this configuration is supported, should the SSDs be visible to the BlueField Arm side as local PCIe/NVMe devices when the auxiliary interface is configured as downstream ports?

The goal is to attach local PCIe/NVMe SSDs behind the BlueField-3 DPU, preferably visible to the BlueField Arm cores, for a storage/JBOF-style prototype.

Any guidance on the supported topology, required adapter hardware, and limitations would be appreciated.

Thanks.

2

Hi pengyu.l,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

The official BF3 Auxiliary Kit (OPN 930-9DAX5-0015-000, Gen5, 150mm cables) uses I-PEX 60-pin Cabline CA-II Plus connectors (P/N 20790-060E-01) — not a PCIe card-edge connector.

There is currently no documentation detailing support for this use case.

Recommend reaching out to our sales and solutions team for guidance surrounding your intended use case: Contact NVIDIA Sales Representatives | NVIDIA

Many thanks,
NVIDIA Enterprise Experience