4x sparks - 2D parallelism (topology) question

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Has anyone with 4x sparks tried running 2D parallelism (TPxPP)?

That would require using Tensor Parallelism (TP) over CX-7, and Pipeline Parallelism (PP) over the RJ-45 cable?

Pipeline Parallelism requires much less bandwidth, since we are splitting the model at a boundary, whereas Tensor Parallelism shards per layer.

Has anyone tested this? Curious about the PP bubbles… and for those of us too cheap to buy a fancy network switch…

3

PP may require less bandwidth, but it’s still very sensitive to latency, and latency on a regular ethernet will likely undo any potential performance gains.

4

I wonder if the USB-C port on the DGX Spark can be used for pipeline parallelism? For example, using a USB-C switch (not sure if such a device exists) for pipeline parallelism?

5

Just get DAC cables for the missing links and you have a 200G ring built.