Hello,
As these are similar, but not identical machines:
can anyone who already tried that comment on possible problems with regard to e.g. differences in firmwares (board firmware, ConnectX-7 firmware, thermal, EC…).
Firmware differences in the NIC might be problematic in particular due to the, as NVIDIA themselves put it, “atypical” setup they chose on the DGX Spark:
Each QSFP port provides 200 Gbps, but to achieve this speed the systems use an atypical setup of two PCIe 5.0 x4 links instead of a single PCIe 5.0 x8 link.
In a typical server design, the ConnectX-7 bridges a QSFP port to a single PCIe 5.0 x8 link to deliver the full 200 Gbps from the QSFP port through to the rest of the hardware.
However, the DGX Spark and GB10 form factor requires swapping a single x8 link for two PCIe 5.0 x4 links to deliver the desired 200 Gbps connection to the Grace-Blackwell SoC.
This setup is not standard. Each QSFP port appears as a pair of Linux network interfaces, with corresponding Remote Direct Memory Access over Converged Ethernet (RoCE) devices. The layout can be confusing the first time you encounter it, and proper configuration can be error-prone.
See Cluster Assistant for ConnectX-7 Multi-Node Clusters
This could mean that firmware affects whether the interconnect behaves as the expected 200 Gb/s-class RoCE link or silently degrades into something slower.
What are the risks here? My use case is mainly Agentic AI workloads with LLMs to be served via vLLM.
Thanks!