# RoCE / RDMA traffic not passing between two Mellanox 200Gbps NIC servers despite link up

**URL:** <https://forums.developer.nvidia.com/t/roce-rdma-traffic-not-passing-between-two-mellanox-200gbps-nic-servers-despite-link-up/355903>\
**Category:** DGX Spark / GB10\
**Created:** [December 29, 2025, 5:14am UTC](https://forums.developer.nvidia.com/t/roce-rdma-traffic-not-passing-between-two-mellanox-200gbps-nic-servers-despite-link-up/355903 "2025-12-29T05:14:23Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\
**Post date:** [December 29, 2025, 6:45am UTC](https://forums.developer.nvidia.com/t/roce-rdma-traffic-not-passing-between-two-mellanox-200gbps-nic-servers-despite-link-up/355903/2 "2025-12-29T06:45:36Z")

</div>

Just follow the original playbooks:

> **[Connect Two Sparks | DGX Spark](https://build.nvidia.com/spark/connect-two-sparks/stacked-sparks)**
>
> Connect two Spark devices and setup them up for inference and fine-tuning

> **[NCCL for Two Sparks | DGX Spark](https://build.nvidia.com/spark/nccl)**
>
> Install and test NCCL on two Sparks

Before anything you should test with just 1 NIC on each Spark and 1 QSFP cable (connect the same port on each spark) and Give it /24 IPs on both Sparks, Verify ping works bidirectionally.

This instruction work fine for 2 cables, but it’s not really needed:

> **[Connecting Two Sparks](https://www.linkedin.com/pulse/connecting-two-sparks-andrew-myers-p9gme?utm_source=share&utm_medium=member_ios&utm_campaign=share_via)**
>
> Stacking two Sparks lets you push into model sizes and throughput a single unit can’t comfortably handle. With two GB10 systems you can theoretically run data/model/tensor parallelism across nodes and pool working memory across processes.

Please take a look at this thread if needed: [ConnectX-7 NIC in DGX Spark](https://forums.developer.nvidia.com/t/connectx-7-nic-in-dgx-spark/350417?page=2)

---

_[View the full topic](https://forums.developer.nvidia.com/t/roce-rdma-traffic-not-passing-between-two-mellanox-200gbps-nic-servers-despite-link-up/355903)._