ConnectX-7 NIC in DGX Spark

To confirm, both 200G physical ports are sharing two PCIe Gen5 x4 lanes.

The theoretical max of a single x4 lane is around 126 Gbit/sec. The theoretical max of using both interfaces is 252 Gbit/sec. That is sufficient to not bottleneck a single 200G interface. Using both 200G interfaces gets you the extra 52 Gbit/sec and pushes the bottleneck back to the PCIe interconnect.

So it’s really up to you to decide whether or not the extra 52 Gbit/sec is worth the second cable.

That summary seems to mirror what I’ve seen with the nccl-test with one cable vs. two cables plugged in.

Does that seem correct?