Do all CX8 network cards support PCIe switch functionality, or is it only the 900-9X81E-00EX-DT0 that supports it?

[mooyu]

Do all CX8 network cards support PCIe switch functionality, or is it only the 900-9X81E-00EX-DT0 that supports it?

[ipavis]

Hi there Mooyu!

Thanks for asking about the PCIe switch functionality in CX8 network cards.
Not all CX8 network cards support PCIe switch functionality. This feature is a specific one that’s only available on certain models.

The 900-9X81E-00EX-DT0 is the one that has the PCIe switch capability. This model supports “Down Stream Port (DSP)” extension option, which essentially turns it into a PCIe root complex that can manage multiple downstream devices like SSDs or GPUs.

Other CX8 models (like the 900-9X81Q-00CN-ST0 and 900-9X81E-00EX-ST0) support different extension options - they use Socket-Direct/Multi-Host capabilities instead, which is great for dual-socket servers but doesn’t give you the PCIe switch functionality.

If you need to connect multiple storage devices or want the CX8 to act as a PCIe switch, you’ll want the DT0 model
If you’re just looking for high-performance networking with dual-socket support, the other models will work great

Hope this helps clarify things!

[mooyu]

Hi!

Thank you for your reply.

For the ConnectX-8 (CX8) NIC:

• On the DT0 version, are the 48 PCIe lanes fixed as 16 lanes upstream (to the host) and 32 lanes downstream (via MCIO for expansion)?

• On the ST0 version, are all 48 lanes used as upstream lanes for multi-host connectivity?

• And is this lane direction (upstream vs downstream) fixed by design and cannot be changed later?

Additionally, if I am using a PCIe 5.0 server, does it mean I can only fully utilize the NIC bandwidth by using the ST0 version together with the auxiliary card, since the DT0 version would be limited by the single x16 upstream link?

[ipavis]

Hi there!

Based on what I’ve checked:

DT0 vs ST0 Lane Configurations
DT0 Version (PCIe Switch Mode):

Yes, exactly! The DT0 has its 48 PCIe lanes locked in as 16 lanes upstream (to your host) and 32 lanes downstream (through MCIO for expansion). Think of it as a PCIe “hub” that can connect multiple devices.

ST0 Version (Multi-Host Mode):
You got it right again! The ST0 uses all 48 lanes as upstream lanes, perfect for connecting multiple hosts to the same NIC.

Can You Change the Lane Direction?
Nope, this is set in stone. The lane direction is hardwired during manufacturing - you can’t reconfigure a DT0 to work like an ST0 or vice versa. It’s like having two different tools designed for different jobs.

PCIe 5.0 Bandwidth Considerations

On a PCIe 5.0 server:
DT0 Version: You’re capped at ~64 GB/s (x16 upstream), which definitely bottlenecks the NIC’s full potential
ST0 Version: This gives you the full bandwidth utilization, especially when paired with the auxiliary card
Bottom Line
If you’re after maximum bandwidth on PCIe 5.0, go with the ST0 version. The DT0 is great if you need PCIe expansion capabilities, but it’s not the choice for maxing out your bandwidth.

Thanks,

Ilan