Connect multiple dxgs

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Good morning everyone,

I bought two nvidia sparks and connected both using QSFP28 cable and followed the guideline on the website but failed to detect cables

$ ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

I connect two cables for both ports

IMG_8624
IMG_86241920×2560 635 KB

2

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, it appears that you have two DGX Spark systems connected using QSFP28 cables, and you are trying to detect the cables using the ibdev2netdev command. However, the output shows that the ports are down.

To troubleshoot this issue, I would recommend checking the following:

  1. Ensure that the QSFP28 cables are properly seated and connected to the correct ports on both DGX Spark systems.
  2. Verify that the QSFP28 cables are compatible with the DGX Spark systems and are not damaged.
  3. Check the DGX Spark system’s configuration to ensure that the InfiniBand ports are enabled and configured correctly.
  4. Run the ibstatus command to check the status of the InfiniBand ports and verify that they are not in a fault state.
  5. If you are using a switch, ensure that it is configured correctly and that the DGX Spark systems are connected to the correct ports.

If you are still having issues, please provide more details about your configuration and setup, and I’ll do my best to assist you further.

3

There are two supported cables.

  • Amphenol: NJAAKK-N911 (QSFP to QSFP112, 32AWG, 400mm, LSZH)
  • Luxshare:- LMTQF022-SD-R (QSFP112 400G DAC Cable, 400mm, 30AWG)

The one in the image is a QSFP28 vs the supported cables that use QSFP112.

This option should also work: https://www.amphenol-cs.com/product/njaakk0006.html

4

Is there any reason why the official cable is QSFP112 and not QSFP56, since the port is limited to 200G anyway?
Officially supported cables are out of stock everywhere (and I couldn’t even find the Luxshare one online), and other QSFP112 options are either backordered or overpriced. Any downside of using QSFP56 ones? Looks like people are getting 200G speeds out of them, according to the posts on this forum.

5

As I’ve posted in Suggested cable to link two Sparks? - #7 by PrinceHal (and as @eugr knows) I have the NADDOD cable (QSFP56) and it seems to work fine. They are marketing it as Spark compatible. Using nccl-test I get about 20GB/sec speeds (expected for 200Gb/sec link).

6

i will try to get other cable and try again since most of cables are out of stock here in germany

7

My assumptions would be the power consumption. And testing a wide range of cables is difficult. But when looking at the specs for the cables, I believe, that the short cable (0.5m) QSFP56 has the same power consumption as the supported cables. Because the power consumptions are similar the thermal impact of the cable should be within spec of the Spark.