# Suggested cable to link two Sparks? **URL:** \ **Category:** DGX Spark / GB10\ **Created:** [October 19, 2025, 1:24pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207 "2025-10-19T13:24:46Z")\ **Posts on this page:** 18\ **Page:** 4
**Author:** ![maiia](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/maiia/32/439026_2.png) [@maiia](https://forums.developer.nvidia.com/u/maiia)\ **Post date:** [November 22, 2025, 7:15pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/61 "2025-11-22T19:15:40Z")
and here’s one more from FS > **[0.5m (2ft) NVIDIA/Mellanox MCP1650-V00AE30 Compatible 200G QSFP56 Ethernet...](https://www.fs.com/products/115634.html)** > > 0.5m (2ft) NVIDIA/Mellanox MCP1650-V00AE30 Compatible 200G QSFP56 Ethernet Passive Direct Attach Copper Twinax Cable, Product Specification:NVIDIA/Mellanox Ethernet Compatible - MCP1650-V00AE30 , Vendor Name - FS , Form Factor - QSFP56 - QSFP56 ,... ---
**Author:** ![e.moe](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@e.moe](https://forums.developer.nvidia.com/u/e.moe)\ **Post date:** [November 22, 2025, 8:17pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/62 "2025-11-22T20:17:43Z")
So when I ordered my cable from NADDOD, I hadn’t discovered this post yet. Not knowing what to get I ordered the MCP1650-H00AE30 instead of the MCP1650-V00AE30. I assumed I wanted the Infiniband cable. I’ve been using it. It does seem to work. I think physically they are the same cable, but there is different identifier in the EEPROM. I don’t know if NADDOD maybe “bins” them differently. I think Infiniband assumes a lossless fabric, where ethernet can handle packet losses better. According to Gemini the DGX Spark is shipped and architected to use **Ethernet** for its clustering fabric, specifically **RoCE v2 (RDMA over Converged Ethernet)**. The ConnectX-7 is a VPI (Virtual Protocol Interconnect) card, meaning it can be switched to InfiniBand mode. However, this is not a simple toggle for the Spark platform. InfiniBand requires a **Subnet Manager (SM)** to be running on one of the nodes to manage Local IDs (LIDs) and route traffic? It would be nice NVIDIA would document this fully. I guess they assume that DGX Spark owners are already familiar with the technology. The only odd thing I see are these warn messages: (EngineCore\_DP0 pid=1896) (RayWorkerWrapper pid=2112) [2025-11-22 05:14:24] spark1:2112:4122 [0] p2p\_plugin.c:421 NCCL WARN NET/IB : roceP2p1s0f0:1 unknown event type (18) Otherwise I think it’s working: (EngineCore\_DP0 pid=1896) (RayWorkerWrapper pid=2112) spark1:2112:7697 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [November 22, 2025, 9:11pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/63 "2025-11-22T21:11:47Z")
Granted, I’m not an expert in Infiniband stack, but Spark seems to use IB connection just fine: ```auto $ ib_write_bw 192.168.177.12 -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB --------------------------------------------------------------------------------------- RDMA_Write BW Test Dual-port : OFF Device : rocep1s0f1 Number of qps : 4 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 128 CQ Moderation : 1 Mtu : 1024[B] Link type : IB Max inline data : 0[B] rdma_cm QPs : ON Data ex. method : rdma_cm --------------------------------------------------------------------------------------- local address: LID 0000 QPN 0x03ec PSN 0xb680ae local address: LID 0000 QPN 0x03ed PSN 0x808800 local address: LID 0000 QPN 0x03ee PSN 0x5b694a local address: LID 0000 QPN 0x03ef PSN 0xe2efd1 remote address: LID 0000 QPN 0x03eb PSN 0x75f6ee remote address: LID 0000 QPN 0x03ec PSN 0x436140 remote address: LID 0000 QPN 0x03ed PSN 0x81698a remote address: LID 0000 QPN 0x03ee PSN 0x4a8b11 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 65536 20000 111.72 111.71 0.213070 --------------------------------------------------------------------------------------- ``` ---
**Author:** ![elsaco](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@elsaco](https://forums.developer.nvidia.com/u/elsaco)\ **Post date:** [November 22, 2025, 10:47pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/64 "2025-11-22T22:47:57Z")
For the DGX Spark the `QSFP56` cable is the best option. The `QSFP112` is for the 400Gbps links and won’t help the Spark squeeze any more bits since it’s capped at 200Gbps system wide. I’m using this cable from FS and works very well: ![QSFT_cable](https://global.discourse-cdn.com/nvidia/original/4X/6/c/8/6c8c0afd66cc1c01f8fd3b622bf6971225051984.jpeg) ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [November 23, 2025, 2:24am UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/66 "2025-11-23T02:24:42Z")
Where did you buy NJAAKKR-0006, @eugr ? Amphenol’s website? $42 at Digikey, but looooooooong lead time. > **[NJAAKR-0006 | DigiKey Electronics](https://www.digikey.com/en/products/detail/amphenol-cs-high-speed-io/NJAAKR-0006/26695994)** > > Order today, ships today. NJAAKR-0006 – 38 Position QSFP Plug to Plug 1.64' (500.00mm) Black Shielded from Amphenol CS (High Speed IO). Pricing and Availability on millions of electronic components from Digi-Key Electronics. ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [November 23, 2025, 4:29am UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/67 "2025-11-23T04:29:13Z")
ExxactCorp. ---
**Author:** ![PrinceHal](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/princehal/32/451063_2.png) [@PrinceHal](https://forums.developer.nvidia.com/u/PrinceHal)\ **Post date:** [November 23, 2025, 5:48pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/68 "2025-11-23T17:48:49Z")
I get the same warning messages with the NADDOD MCP1650-V00AE30 cable, if it makes you feel any better. ---
**Author:** ![PrinceHal](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/princehal/32/451063_2.png) [@PrinceHal](https://forums.developer.nvidia.com/u/PrinceHal)\ **Post date:** [November 23, 2025, 5:50pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/69 "2025-11-23T17:50:12Z")
Laser? Is that an active optical cable? Ours are copper. ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [November 23, 2025, 6:45pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/70 "2025-11-23T18:45:24Z")
Hey @eugr and @maiia, How’s the experience so far with the spark cluster, what type of workloads have you been experimenting? Any testing with distributed training yet? ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [November 24, 2025, 12:54am UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/71 "2025-11-24T00:54:34Z")
A bit frustrating, to be fair. I’d appreciate a bit more clear guidance and participation from NVidia on this. For instance, whether we need to bond both interfaces for a physical port together to get 200G speeds, because I’m not able to get more than 100G with iperf3 or ib\_write\_bw. Other frustrations are mostly related to the state of Blackwell support in VLLM and the worst mmap performance on any system I’ve used recently. When it takes 10 minutes to just load a 80GB model into vllm, it results in a lot of wasted time. Haven’t done anything useful in terms of clustering so far… ---
**Author:** ![maiia](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/maiia/32/439026_2.png) [@maiia](https://forums.developer.nvidia.com/u/maiia)\ **Post date:** [November 24, 2025, 2:52am UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/72 "2025-11-24T02:52:34Z")
We’re capped by the DGX Spark Physical PCle Link Hardware. This is from Gemini 3 Pro: ```bash sudo lspci -vvv | grep -A 20 "NVIDIA\|Mellanox" | grep -E "LnkCap|LnkSta" ``` ```auto LnkCap: Port #0, Speed 32GT/s, Width x4, ASPM not supported LnkSta: Speed 32GT/s, Width x4 LnkCap: Port #0, Speed 32GT/s, Width x4, ASPM not supported LnkSta: Speed 32GT/s, Width x4 LnkCap: Port #0, Speed 32GT/s, Width x4, ASPM not supported LnkSta: Speed 32GT/s, Width x4 LnkCap: Port #0, Speed 32GT/s, Width x4, ASPM not supported LnkSta: Speed 32GT/s, Width x4 pcilib: sysfs_read_vpd: read failed: No such device LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L1, Exit Latency L1 <4us ``` **PCIe Link Speed:** Each Mellanox/NVIDIA NIC shows “Speed 32GT/s, Width x4” (PCIe Gen5 x4). The expected is x16 for full-bandwidth. Gen5 x4 maxes out at ~16 GB/s per lane, and only ~64 Gbps per cable—not sufficient for your 200G physical link, which explains the observed effective speed ceiling. I wonder if NVIDIA will review BIOS/firmware options to enhance PCIe slot configuration bottleneck. I can’t believe I’m spending this long in device setup before getting to work on anything meaningful. ---
**Author:** ![elsaco](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@elsaco](https://forums.developer.nvidia.com/u/elsaco)\ **Post date:** [November 24, 2025, 3:30am UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/73 "2025-11-24T03:30:08Z")
When a cable connector is inserted in one of the physical ports two virtual links light-up, i.e. `enp1s0f0np0` and `enP2p1s0f0np0` Each virtual link is x4 wide capable of 32GT/s, so the combined output will be x8 and 64GT/s speed. That’s enough to send, theoretically, 200Gbps on the wire! Even the `Vital Product Data` claims it: `Product Name: NVIDIA DGX Spark, P4242-0000, 2-port QSFP up to 200G, Ethernet, PCIe5` Use `lspci -vvvs ` for details. ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [November 24, 2025, 3:34pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/74 "2025-11-24T15:34:57Z")
This cable started selling at Microcenter: [https://www.microcenter.com/product/703095/pny-certified-stacking-dgx-spark-cable-40](https://www.microcenter.com/product/703095/pny-certified-stacking-dgx-spark-cable-40) Theoretically, the official certified cable for stacking DGX sparks by PNY. ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [November 24, 2025, 6:28pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/75 "2025-11-24T18:28:01Z")
That’s a longer version, NJAAKK-N911. The one they had before was NJAAKR one. ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [November 24, 2025, 7:05pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/76 "2025-11-24T19:05:10Z")
It says 1.64 ft. (0.50 m) for this one. The previous one says (NJAAKK-0006): usage: Connector the new one (NJAAKK-N911): Usage: Stacking On PNY’s website, they list both ![image](https://global.discourse-cdn.com/nvidia/original/4X/5/3/0/530a5caa79950a797cf627c91dfb1d3120fe5d39.jpeg) :) Confusing ---
**Author:** ![elsaco](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@elsaco](https://forums.developer.nvidia.com/u/elsaco)\ **Post date:** [November 24, 2025, 7:49pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/77 "2025-11-24T19:49:38Z")
The only difference is cable length: NJAAKK-N911 = 400mm (0.4 meters) NJAAKK-0006 = 500mm (0.5 meters) Both cables are made by Amphenol. The part after NJAAKK signifies the cable length. See details at [https://www.amphenol-cs.com/catalogsearch/result?query=NJAAKK&page=1](https://www.amphenol-cs.com/catalogsearch/result?query=NJAAKK&page=1) ---
**Author:** ![BruBoy](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@BruBoy](https://forums.developer.nvidia.com/u/BruBoy)\ **Post date:** [November 24, 2025, 8:08pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/78 "2025-11-24T20:08:25Z")
Thanks for this suggestion. I struggled to find a suitable cable available here in the UK but the NADDOD one arrived quickly from China and is working for me: [NVIDIA/Mellanox 0.5m 200G QSFP56 Passive Direct Attach Cable for DGX Spark (Powered by the GB10 Grace Blckawell Superchip) Dual-System Interconnect - NADDOD](https://www.naddod.com/products/13888.html) ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [December 8, 2025, 8:08pm UTC](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207/79 "2025-12-08T20:08:31Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed. [Previous page](https://forums.developer.nvidia.com/t/suggested-cable-to-link-two-sparks/348207.md?page=3)