Slow speeds on DGX spark cluster

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi All,

So I decided to buy 2 dgx sparks and a mikrotik CRS804 DDQ to start my DGX Spark cluster. I want to scale beyond 2 in the near future so this is why I decided to start with a switch. I followed the Nvidia guide but cant get is to work. The NCCL test wont go higher than 0.5GB/s.

I use a FS 400g Breakout cable (QDD-400G-2QPC02)

Summary — Two DGX Spark (GB10) units through a MikroTik CRS804-4DDQ (400G→2×200G breakout). Link negotiates 200 Gb/s and is functionally healthy (ping/TCP/NCCL all work, #wrong 0), but RDMA/NCCL throughput is stuck at ~0.5 GB/s where others report 16–23 GB/s. Raw ib_write_bw ~0.5–0.7 Gb/s while ib_write_lat is a healthy ~7.5 µs, and HW counters show packet_seq_err + rp_cnp_handled climbing every transfer (DCQCN throttling) despite zero fabric drops.

2

Power off everything, unplug power from wall, wait a minute and power it back up.

The CX-7 needs a full power drain after detecting cables before the correct settings are applied.

Pretty much everyone seems to run into this issue on the first use after DAC cables are plugged in.

3

Open grok, register, use it on terminal “grok” CLI, make him fix it, then learn when he explains , be happy!!!

4

I have been using claude code to help me fix this. And it validated everything that I did already. It suggested powering down the Sparks for at least a minut. But since I am logging in remotely I am not able to do that.

@mashie basicly suggested the same as claude code so I will do that as soon as possible.

5

@Thom.S you could try using the script in /opt/nvidia/dgx-spark-mlnx-hotplug folder to remove the CX-7 and add it back. This will reset your network interface:

sudo /opt/nvidia/dgx-spark-mlnx-hotplug/mtk-hotplug-handler.sh removal

sudo /opt/nvidia/dgx-spark-mlnx-hotplug/mtk-hotplug-handlers plug-in

It might fix your issue. This script only works on the CX-7 interface so your management link won’t be affected.

6

You just need the power drain, it is that simple. The CX7 will not update its firmware settings without it.

7

MikroTik interfaces are set to auto-negotiate by default, but I had to explicitly set the bandwidth to get ~20-24GB/sec (4x cluster)

8

I have the same switch, connected to 4 DGX Sparks using QSFP-56 DAC cables, and the connection works at full speed. Auto-negotiation turned off, set to 200G baseCR4 manually. I also have the networking on the sparks manually configured, here’s an example from one of them:

network:
  version: 2
  ethernets:
    enp1s0f0np0:
      addresses:
        - 10.10.0.10/24
      dhcp4: no
      mtu: 9000
    enP2p1s0f0np0:
      addresses:
        - 10.10.1.10/24
      dhcp4: no
      mtu: 9000
    enp1s0f1np1:
      addresses:
        - 10.10.128.30/24
      dhcp4: no
      mtu: 9000
    enP2p1s0f1np1:
      addresses:
        - 10.10.129.30/24
      dhcp4: no
      mtu: 9000

Notice the logical interfaces for each CX-7 port are on different subnets. Back on the switch, the bridge is on 10.10.0.1/8. I’m far from an expert on this but it’s working for me.

9

It’s in the RouterOS manual:

Link Configuration:

400G: Can be configured with either auto-negotiation or a forced 200G speed.
2x200G, 4x100G, 8x50G,  8x25G, 2x40G, 8x10G and 8x1G: Must be set with a forced speed mode and auto-negotiation disabled.

When splitting a 400G port on the CSR804 the speed must be explicitly set!

10

@elsaco I did that already.

@amurnane123 Thanks for the settings. I just verified that mine are the same. Will try and powerdrain the sparks tomorrow.

11

Hi all,

I have done the powerdrain. But it did not work.

Some more information about the settings:

I plugged the cable in port 1 of the switch. And the other side in port 1 of spark 1 and in port 1 of spark 2.

Al settings:

Platform

Column 1 Column 2 Column 3
Item thor loki
Model / GPU NVIDIA DGX Spark — GB10 NVIDIA DGX Spark — GB10
OS Ubuntu 24.04.4 LTS (aarch64) Ubuntu 24.04.4 LTS (aarch64)
Kernel 6.17.0-1026-nvidia 6.17.0-1026-nvidia
CX-7 firmware 28.45.4028 28.45.4028
PCIe-link CX-7 32,0 GT/s (Gen5) ×4 vol 32,0 GT/s (Gen5) ×4 vol
NCCL / nccl-tests 2.28.9+cuda13.0 / 2.19.6 2.28.9+cuda13.0 / 2.19.6
Open MPI 4.1.6 4.1.6

Interfaces

Column 1 Column 2 Column 3 Column 4 E F
Interface Rol thor loki MTU Status
enp1s0f0np0 RoCE rail 1 192.168.100.1/24 192.168.100.2/24 9000 UP 200G
enP2p1s0f0np0 RoCE rail 2 192.168.101.1/24 192.168.101.2/24 9000 UP 200G
enP7s7 Management 10.20.221.227/28 10.20.221.226/28 1500 UP
tailscale0 Remote 100.96.70.114 100.123.179.8 — UP
enp1s0f1np1 2e port, not used — — — DOWN
enP2p1s0f1np1 2e port, not used — — DOWN

RDMA and QoS

Setting Value
RDMA-devices rocep1s0f0, rocep1s0f1, roceP2p1s0f0, roceP2p1s0f1
Poortstatus 4: ACTIVE · 5: LinkUp · 200 Gb/sec (4X HDR)
GID-index 3 RoCE v2, IPv4 — correct for NCCL_IB_GID_INDEX=3
Trust state dscp
PFC enabled 0 0 0 1 0 0 0 0 — prioriteit 3
DSCP→prio dscp 24–31 → prio 3 (so dscp 26 works)

Switch

Item Value
Switchchip Marvell-98DX7335 — qos-hw-offloading: yes, l3-hw-offloading: no
RouterOS 7.23.2 (stable), build 2026-07-03 — factory 7.19.4
Bridgepoorten qsfp56-dd-1-1 (thor), qsfp56-dd-1-5 (loki) — hw-offload: yes
Bridge bridge1 Spark cluster — actual-mtu 9000, l2mtu 9016, rstp, vlan-filtering: no
Breakout 400G → 2×200G
Management-IP 10.20.221.225 — seperate switch
Parameter Value Note
auto-negotiation no must be off when speed is forced
eeprom-checksum bad known why auto-neg must be off; not a fault source
fec fec91 fec-mode auto
mtu / l2mtu 9000 / 9016 jumbo, matches the NICs
sfp-type QSFPDD, present FS QDD-400G-2QPC02 breakout
speed 200G-baseCR4 forced, as the manual requires with breakout
status link-ok, full-duplex rate 200Gbps
tx/rx-flow-control off correct — PFC replaces global pause

QOS - Lossless ROCE

Parameter Value
pfc pfc-tc3 — on both ports
egress-rate-queue3 200.0Gbps required for PFC Rx
trust-l2 / trust-l3 ignore / keep
profile roce dscp 26 → traffic-class 3, hw-offloaded
profile default dscp 0 → traffic-class 1, hw-offloaded

Schermopname 30-07-
Schermopname 30-07-1001×566 76.3 KB

It seems like the switch is throwing away 75% of its traffic without any error counters.

The measurement

Baseline of both NICs and both switch ports, one test, second reading. --tclass=104 sets DSCP 26 so the traffic lands in the same lossless class NCCL uses — without it ib_write_bw defaults to DSCP 0 and you measure the wrong queue.

Server on thor, client on loki:

ib_write_bw -d rocep1s0f0 -x 3 -F --report_gbits --tclass=104 -s 65536 -n 20000 [192.168.100.1]

Result: 0.75 Gb/s average, 5.13 Gb/s peak.

Stage Bytes
Useful payload (20,000 × 64 KiB) 1,310,720,000
loki put on the wire (tx_prio3_bytes) 44,814,630,472
switch ingress from loki (rx-bytes) 44,814,862,100
switch egress to thor (tx-queue3-byte) 10,976,973,864
thor received (rx_prio3_bytes) 10,976,973,864
Lost inside the switch 33,837,656,608 — 75.5%

Switch egress and thor’s receive counter agree to the byte, and loki’s transmit matches the switch ingress to within 0.0005%. The traffic genuinely disappears between the two switch ports.

The ACK direction is hit too: thor sent 3,269,838 bytes, loki received 911,394 — 72% lost. That is what turns packet loss into a collapse. The sender retransmits go-back-N, so 1.31 GB of payload costs 44.81 GB of wire traffic (34× amplification), leaving 0.75 Gb/s.

NIC counters: thor out_of_sequence 3,770; loki roce_adp_retrans 7,333, packet_seq_err 463.

Why this stayed hidden

Throughout, on both switch ports:

tx-drop-queue0..7 (packet and byte):  0

tx-drop-packet / tx-drop-byte: 0

rx-overflow / rx-fcs-error: 0

rs-fec-uncorrected: unchanged

And on both NICs: rx_prio3_pause, tx_prio3_pause, rx_prio3_pause_transition all 0.

33.8 GB discarded, nothing recorded it, and PFC never fired despite the fabric plainly failing to deliver. The physical layer is clean, so this is not a PHY issue. Every error counter reading zero is why the switch looked innocent for weeks.

12

So what is the exact topology you are testing now:

Two sparks on the same splitter DAC going into one switch port, or two sparks on individual splitter DACs going into different ports on the switch?

Move the DACs around and see if the error direction moves with them.

Also do you see the same loss on the switch if you send traffic in the reverse direction?

13

I use the https://www.fs.com/nl/products/182807.html

The splitted ends are connected to a spark each. So they share a port on the switch.

I have tested the traffic going each way. I already moved over the ports on the switch and the sparks.