RoCE / RDMA traffic not passing between two Mellanox 200Gbps NIC servers despite link up

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

RDMA _ RoCE Connectivity Issue Between Two GPU Servers (Ping & ib_write_bw Failing).pdf (121.3 KB)

Hi,
I need help on connecting two DGX spark GB10 server.

Two GPU servers connected via NVIDIA/Mellanox 200 Gbps NICs show link UP and RDMA devices ACTIVE, but no IP connectivity (ping) and RDMA tests (ib_write_bw) fail between the hosts on the RoCE interfaces. Management network works normally.

🔹 Issues / Problem Statement

  1. Physical link is UP at 200 Gbps on both servers
  2. rdma link shows ACTIVE / LINK_UP for all RoCE devices
  3. Ping fails between servers on RDMA IP subnets
  4. ib_write_bw fails with socket connection error
  5. Correct IP addressing and routing present
  6. MTU tested with 1500 (no jumbo frames)
  7. Issue persists across all RDMA interfaces

🔹 Impact

  1. Unable to validate or use RDMA / RoCE between GPU servers
  2. Blocking GPU workload performance testing and deployment

🔹 Environment (Short)

  1. OS: Ubuntu Linux
  2. NIC: NVIDIA/Mellanox 200 Gbps (DAC, back-to-back)
  3. RDMA Type: RoCE
  4. Management network: 10.3.3.0/24 (working)
  5. RDMA networks: 192.168.100.0/24 – 192.168.103.0/24 (not working)
2

Just follow the original playbooks:

Before anything you should test with just 1 NIC on each Spark and 1 QSFP cable (connect the same port on each spark) and Give it /24 IPs on both Sparks, Verify ping works bidirectionally.

This instruction work fine for 2 cables, but it’s not really needed:

Please take a look at this thread if needed: ConnectX-7 NIC in DGX Spark

3

Hi @raphael.amorim , i have followed all the steps you have provided but not able to ping also
Could you please verify my connection

image
image512×339 71.6 KB

4

could you please connect port 1 with port 1 and port 2 with port 2. Connecting like this won’t work. And start with a single cable, you don’t need 2 cables at all with 2 sparks.

5

@raphael.amorim Thanks, solved the issue