QSFP interface available on one Spark but not the other

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi,

I’m following the official Connect Two Sparks playbook. At step 2, I can see the interfaces are available as expected on spark-1:

hakeem@spark-1:~$ 
ibdev2netdev 
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up) 
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down) 
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up) 
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

However, one isn’t available on spark-2:

hakeem@spark-2:~$ 
ibdev2netdev 
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down) 
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up) 
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

Hence when I move to step 4, spark-2 is not recognized.

Additionally, the QSFP interfaces on spark-1 only appear properly when the cables are on the left-most ports of both spark-1 and spark-2. Any other arrangement results in all ports being Down, or otherwise unrecognized.

How can I solve this?

Thank you.

2

Is there anything in sudo dmesg that would suggest a problem with the device / device driver? (you can compare between the devices since one is working as you expect.)

What is in your related /etc/netplan/* files?

Did you try a reboot?

3

Thanks!

I used lspci to find the controllers, and found that the driver module mlx5_core wasn’t bound properly. Unplugging and replugging the Spark fixed it.

1 Like