Can't stack DGX Sparks - HELP

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I’ve followed the NVIDIA guide step by step so many times through i’ve lost count. Seems promising, IP addresses fix, can ping eachother, share SSH keys, but NCCL tests fail and when i did get throughput it was less than 1gbs. Just absolutely lost in the sauce, not sure if I should nuke from orbit? I could really use a pointer or two thanks in advance.

2

What cable are you using?

3

Can you provide nvidia-bug-report and the NCCL run logs?

4

I bought the official Amphenol cable directly from NVIDIA

5

mpirun -np 2 -H 192.168.100.10:1,192.168.100.11:1

–mca plm_rsh_agent “ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no”
-x LD_LIBRARY_PATH=$LD_LIBRARY_PATH
$HOME/nccl-tests/build/all_gather_perf
Warning: Permanently added ‘192.168.100.10’ (ED25519) to the list of known hosts.

WARNING: An invalid value was given for btl_tcp_if_include. This
value will be ignored.

Local host: spark1

Value: enp1s0f1np1
Message: Unknown interface name

[spark1:21500] [[22715,1],1] selected pml ob1, but peer [[22715,1],0] on unknown selected pml ���

MPI_INIT has failed because at least one MPI process is unreachable
from another. This usually means that an underlying communication
plugin – such as a BTL or an MTL – has either not loaded or not
allowed itself to be used. Your MPI job will now abort.

You may wish to try to narrow down the problem;

  • Check the output of ompi_info to see which BTL/MTL plugins are
    available.
  • Run your application with MPI_THREAD_SINGLE.
  • Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
    if using MTL-based communications) to see exactly which
    communication plugins were considered and/or discarded.

[spark1:21500] *** An error occurred in MPI_Init
[spark1:21500] *** reported by process [1488650241,281470681743361]
[spark1:21500] *** on a NULL communicator
[spark1:21500] *** Unknown error
[spark1:21500] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[spark1:21500] *** and potentially your MPI job)

6

?? help ??

7

I have written the following little project for testing the connection. You can use it to verify the cable speed and the data dump. The approach is basically I document what is my settings in spark1 and spark2, and let ai to figure out the rest. If I ware you, I would use the same port connection, I mean: the left port of spark1 to connect to left port of spark2), in my connection at the time: right of spark1 to left of spark2.

8

Have a look at eugr’s network documentation:

May this helps to find the missing puzzle piece

Or give sparkrun a try:

It does include a setup wizard.

9

Please run with NCCL_DEBUG=INFO and share the logs from your run as well as nvidia-bug-report. Also share your ip a output from both units.

10

the answer was to plug in CLAUDE CODE and hit yes for a day and a half. Working great now!

11

Correct. :)