I’d like to ask about the port configuration of the ConnectX-7 NIC in DGX Spark. On our DGX Spark, the onboard ConnectX-7 NIC appears as four ports when checking with ip -br a and lspci: roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down) roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up) rocep1s0f0 port 1 …

This is the expected behaviour due to a limitation in the GB10 chip. The SoC can’t provide more than x4-wide PCIe per device, so, in order to achieve the 200gbps speed, we had to use the Cx7’s multi-host mode, aggregating 2 separate x4-wide PCIe links, which combined can deliver the 200gbps speed. …

Thanks for testing. we indeed can saturate a single 200 GbE port over one cable with iperf3 (multi-stream) and with RDMA verbs (ib_write_bw). NCCL shouldnt benefit from balance-rr/LACP: a single NCCL flow/QP hashes to one link, and balance-rr can cause reordering :(

So far I’ve only been able to get just over 18 GB/s across both 200Gb links with NVCC all_gather_perf, with four rail RoCE v2. I’m not sure why you would use LACP or XOR bonding here, since I think getting the lowest latency with RoCE v2 is really important, and using RDMA multirail should provide s…

[image] s.matheny: Something interesting is that RDMA latency by default is around 1-1.5ms between two Sparks, which is really horrible. This seems to be related to the power management C-states that are set to be really sleepy by default on these units. You can’t measure RDMA latency using…

[image] s.matheny: sudo cpupower frequency-set -g performance This one, btw, seems redundant, as “performance” governor is set by default on Sparks. Disabling idle states will likely result in much higher power consumption at idle, though. I wonder if 6.17 kernel addresses it (or it’s schedu…

Thanks for sharing this @s.matheny [image] From 500–980 µs → 47–73 µs a 10–15× improvement in kernel network stack latency. Not much change in RDMA Latency and NCCL tests as expected. [image] [image] [image] [image] [image] [image]

[image] s.matheny: sudo cpupower idle-set -D 0 sudo cpupower frequency-set -g performance Just wanted to say I have a similar speedup here, and it helped boost Llama 3.2:3b performance from 86 to 95 tokens/s inference (prompt processing didn’t see as substantial a boost, but was also improved…

I’m seeing an improvement too: gpt-oss-120b Cluster: build/bin/llama-bench -m ~/.cache/llama.cpp/ggml-org_gpt-oss-120b-GGUF_gpt-oss-120b-mxfp4-00001-of-00003.gguf --rpc 192.168.177.12:15001 -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0 model size params backend test t/s g…

HOWEVER, there is another way that can potentially improve it even more - it was suggested to try libvma, which can be inserted to any process via LD_PRELOAD and offload TCP/IP to Mellanox hardware completely, decreasing latency. I’ve done some research, and it looks like it’s been superseded by li…

ConnectX-7 NIC in DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10

s.matheny November 27, 2025, 7:22am 54

Absolutely right… pings are TCP stack, and I misspoke.

Topic		Replies	Views
NCCL Test Bandwidth is only 3GB/s between 2 DGX Spark using QSFP cable DGX Spark / GB10 spark , nics , dgx	9	481	April 19, 2026
Suggested cable to link two Sparks? DGX Spark / GB10	76	8152	November 24, 2025
Test the sample about "Connect Three DGX Spark in a Ring Topology" DGX Spark / GB10 cuda	15	720	April 13, 2026
Dual DGX Spark RoCE Bandwidth Expectations DGX Spark / GB10	20	879	May 14, 2026
DGX Spark direct QSFP connection only getting ~13-16 Gbps instead of expected 200G performance DGX Spark / GB10	10	487	May 14, 2026
NCCL bandwidth capped at 3 GB/s, GPU PCIe topology reports Gen1 x1 on DGX Spark FE DGX Spark / GB10 pcie , kernel , performance , debugging-and-troubleshooting , nics , rdma	5	367	April 14, 2026
DGX Spark NCCL Test: 10GB/s not 200 Gbps=25 GB/s DGX Spark / GB10	3	931	November 5, 2025
Why is my NCCL broken? DGX Spark / GB10	25	617	February 5, 2026
ConnectX‑7 200GbE via MikroTik CRS812 + QSFP‑DD 400G → 2xQSFP56 200G breakout DGX Spark / GB10	5	1469	January 10, 2026
One of Four DGX Sparks Shows ~35% Lower NCCL Bandwidth — Can't Figure Out Why DGX Spark / GB10	49	1683	March 17, 2026

ConnectX-7 NIC in DGX Spark

Related topics