Just getting started with DGX and jumped from 2 nodes to 4 so now life gets harder networking wise. I purchased the Mikrotik CRS812 DDQ based on community success stories and my plans were 4 nodes and ‘other devices’ so that model fit the bill for me.
For context, I’m a software guy not a network guy. So forgive any ignorance here, I expected to have to poke a few settings here and there but this thing has a crazy complex configuration surface and I’m trying some AI recommended commands but those keep choking and the commands look somewhat legit but this RouterOS deal is new to me and not very intuitive or forgving. I’ve already learned how to reset of course…
My question is - do we as a community have any tried and true / documented playbooks for going from out of box experience to up and going for a 4 node cluster?
I’m aiming to try and document and figure it out as I go so that I can restart/script it as needed long-term and knowing I’ll make mistakes… reset and script is the goal.
Cables in hand, switch arrived but struggling so far - I haven’t just tried things out and used their WinBox GUI as it presents an overwhelming control surface for what looks like every featrure across every product - so I’ve hesitated to just experiment blindly.
Any suggestions on walk-throughs, playbooks, scripts out there? I’ve done a ton of reading of the materials I could find and some really helpful ones with gotchas but I was looking for a true walk-through or better yet - script. Seems like a recipe shoudl exist for the switch side…
Am I missing out on any community resources available for this?
It depends, I have just been in the mode of getting it up and running reliably with the models I want to run. I haven’t done any hardcore optimizations or anything. With the CRS812, you’ll have 2 400Gbe ports. If you only have 200Gbe DAC cables, you’ll need to disable auto-negotiation on those ports to 200G and life will be easier.
The rest is just standard practice, usually just to make sure all nodes can reach each other through SSH without password. Additionaly, make sure netplan defines a static IP for the RoCE interfaces.
Helpful to also set MTU to 9000 in the switch too and set that in your netplan.
That’s the part that will save you a lot of headaches. Tempting to do dynamic IP allocation, but if it changes, NCCL will not be friendly when launching vLLM after a reboot.. Use eugr’s docker image for vLLM gets you all the way there.
I purchases the 2x 200g breakout cables - my understanding is I have to go and manually setup the breakout for many of the must-have settings and a few ‘to optimize it’.
THis is what the Qwen model spit out so far after I kept correcting it on which ports can actually be used in the breakout…
/interface ethernet switch qos port set qsfp56-dd-1-1 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-1-2 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-1-3 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-1-4 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-2-1 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-2-2 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-2-3 egress-rate-queue3=0
/interface ethernet switch qos port set qsfp56-dd-2-4 egress-rate-queue3=0
6. Finalize
/system backup save name=“dgxspark-configured-state”
/export file=“dgxspark-configured-state.rsc”`
My experience was that trying to use the CLI for those suggested commands simply wasn’t working - and at first I figured it was my ignorance of the RouterOS, partly true - but I also didn’t know if the commands were legit or not. At a glance, they look legit to the uninformed (me!).
At this time, I’m using Sparkrun recipes for running / serving models for context, so vllm is used in many of those multi-node recipes. I’d assume sglang and others - when it comes time to configure the Sparks I planned to use Sparkrun and/or the NVidia Sync tool deal.
I think you can ask Claude for commands to output status of each port. But I do it the old fashioned way and just check through the management interface in browser. Sometimes, I am so lazy I just change things through the UI anyway. :)
I’m about to go full experimental mode via GUI. That’s definitely looking like an easier road. I have had Hermes + Qwen exploring via SSH but it struggles with RouterOS. I kept getting an invalid syntax error with all the variants I tried. So GUI to the rescue… I foresee several resets in my future potentially if I botch things. That’s why i was looking for a little more sure footing before just trying things.
I asked the agent to compile some notes, mine the forums, several things I had previously eyeballed learning before makign the purchase.
For the sake of sharing, I’ll show what the agent mined for me from the community resources I could find where these switches are cited as being used. I’ll put the 2 main outputs in replies.
This appendix provides a complete, categorized bibliography of every technical resource, community insight, and official specification utilized to build the MikroTik CRS812 DGX Spark deployment framework. This ensures traceability and allows engineers to perform deep-dives into specific implementation decisions.
Used for: Reference architectures and multi-node connectivity patterns.
🤝 Community Intelligence & Field Implementation
These resources provided the “real-world” intelligence required to overcome hardware quirks (like the Marvell chipset limitations and breakout negotiation failures).
Part 2 of the notes it compiled for the quirks/gothas somewhat spitting back out what another community member had already documented since I cited that as a resource to evaluate so credit due where credit is due there.
This document outlines high-probability failure points encountered during the deployment of 200GbE RoCE v2 fabrics using the MikroTik CRS812. Read this BEFORE troubleshooting hardware.
1. The “No Link” Trap (Layer 1/2)
Symptom: Port LED is off, or ethtool shows Link detected: no.
The Breakout Mismatch: If you are using a 400G $\to$ 2x200G breakout cable, the switch must have breakout-mode=2x200G set on the parent port. Without this, the sub-ports (qsfp56-dd-1-1, etc.) will not even appear in the interface list.
FEC (Forward Error Correction) Mismatch: This is the #1 killer of 200G links.
Requirement: For most 200G DAC/AOC breakout cables, RS-FEC (fec=rs-fec) is mandatory on both the switch and the ConnectX-7 NIC.
Failure Mode: If one side is auto or Base-R FEC and the other is RS-FEC, you may see a link, but with massive CRC errors or zero throughput.
Auto-Negotiation:DISABLE IT. High-speed (200G+) negotiation often fails in mixed-vendor environments. Force both sides:
Switch: /interface ethernet set <port> auto-negotiation=disabled speed=200Gbps
Node: sudo ethtool -s <iface> autoneg off speed 200000 duplex full
2. The “RDMA/NCCL Performance” Trap (Layer 3/4)
Symptom: Link is up at 200G, but NCCL tests show $<10$ GB/s or fall back to TCP.
The GID Index Problem: RoCE v2 requires a specific GID index to be selected for the connection. This index can change after a switch reboot or NIC driver update.
Check: Run rdma link show gid-table on the node.
Fix: Ensure your NCCL environment variable matches: export NCCL_IB_GID_INDEX=<index_found_above>.
MTU Mismatch (The Silent Killer): If any single device in the path (Switch, Node 1, Node 2, or even a virtual bridge) is not set to 9000, RoCE packets will be dropped or fragmented, causing massive performance degradation.
Check:ip link show | grep mtu on all nodes and /interface ethernet print on the switch.
PFC (Priority Flow Control) Missing: Without PFC, RoCE is not “lossless.” If congestion occurs, packets drop, and RDMA performance collapses.
Requirement: Ensure pfc-priority=3 (or your chosen priority) is enabled on the switch and the NICs are configured to honor it.
3. The “Multi-Host” Confusion
Symptom: You see two interfaces (enp1s0f0np0, enp1s0f1np1) but only one cable is plugged in.
Reality Check: ConnectX-7 cards in Multi-Host mode present multiple logical interfaces per physical port. Ensure your configuration treats them as distinct endpoints for your cluster nodes.
4. Summary Checklist for Deployment
Breakout Mode enabled on Parent Port?
Auto-Neg disabled on both Switch and NIC?
Speed forced to 200Gbps on both sides?
FEC set to rs-fec on both sides?
MTU 9000 end-to-end (Switch $\leftrightarrow$ NIC)?
Some of its assumptions are flat out incorrect it appears. I keeps insisting the 400g ports have parents but those appear to be the other 2x 200g for instance. So that’s why I questioned everything it was spewing out. Wasn’t sure if it was AI slop or not.
Here’s what the out-of-box looks like - so I’ve had it probe/inspect things and yet it still seems to get confused. I should be using 10-13 | 14-17 | 18-21 | 22-25, etc.
[admin@MikroTik] /interface> print
Flags: R - RUNNING; S - SLAVE
Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
NAME TYPE ACTUAL-MTU L2MTU MAX-L2MTU MAC-ADDRESS
0 RS ether1 ether 1500 1600 9586 D0:EA:11:67:47:D1
1 S ether2 ether 1500 1600 9586 D0:EA:11:67:47:D2
2 S qsfp56-1-1 ether 1500 1584 9570 D0:EA:11:67:47:C1
3 S qsfp56-1-2 ether 1500 1584 9570 D0:EA:11:67:47:C2
4 S qsfp56-1-3 ether 1500 1584 9570 D0:EA:11:67:47:C3
5 S qsfp56-1-4 ether 1500 1584 9570 D0:EA:11:67:47:C4
6 S qsfp56-2-1 ether 1500 1584 9570 D0:EA:11:67:47:C5
7 S qsfp56-2-2 ether 1500 1584 9570 D0:EA:11:67:47:C6
8 S qsfp56-2-3 ether 1500 1584 9570 D0:EA:11:67:47:C7
9 S qsfp56-2-4 ether 1500 1584 9570 D0:EA:11:67:47:C8
10 S qsfp56-dd-1-1 ether 1500 1584 9570 D0:EA:11:67:47:B1
11 S qsfp56-dd-1-2 ether 1500 1584 9570 D0:EA:11:67:47:B2
12 S qsfp56-dd-1-3 ether 1500 1584 9570 D0:EA:11:67:47:B3
13 S qsfp56-dd-1-4 ether 1500 1584 9570 D0:EA:11:67:47:B4
14 S qsfp56-dd-1-5 ether 1500 1584 9570 D0:EA:11:67:47:B5
15 S qsfp56-dd-1-6 ether 1500 1584 9570 D0:EA:11:67:47:B6
16 S qsfp56-dd-1-7 ether 1500 1584 9570 D0:EA:11:67:47:B7
17 S qsfp56-dd-1-8 ether 1500 1584 9570 D0:EA:11:67:47:B8
18 S qsfp56-dd-2-1 ether 1500 1584 9570 D0:EA:11:67:47:B9
19 S qsfp56-dd-2-2 ether 1500 1584 9570 D0:EA:11:67:47:BA
20 S qsfp56-dd-2-3 ether 1500 1584 9570 D0:EA:11:67:47:BB
21 S qsfp56-dd-2-4 ether 1500 1584 9570 D0:EA:11:67:47:BC
22 S qsfp56-dd-2-5 ether 1500 1584 9570 D0:EA:11:67:47:BD
23 S qsfp56-dd-2-6 ether 1500 1584 9570 D0:EA:11:67:47:BE
24 S qsfp56-dd-2-7 ether 1500 1584 9570 D0:EA:11:67:47:BF
25 S qsfp56-dd-2-8 ether 1500 1584 9570 D0:EA:11:67:47:C0
26 S sfp56-1 ether 1500 1584 9570 D0:EA:11:67:47:C9
27 S sfp56-2 ether 1500 1584 9570 D0:EA:11:67:47:CA
28 S sfp56-3 ether 1500 1584 9570 D0:EA:11:67:47:CB
29 S sfp56-4 ether 1500 1584 9570 D0:EA:11:67:47:CC
30 S sfp56-5 ether 1500 1584 9570 D0:EA:11:67:47:CD
31 S sfp56-6 ether 1500 1584 9570 D0:EA:11:67:47:CE
32 S sfp56-7 ether 1500 1584 9570 D0:EA:11:67:47:CF
33 S sfp56-8 ether 1500 1584 9570 D0:EA:11:67:47:D0
;;; defconf
34 R bridge bridge 1500 1584 D0:EA:11:67:47:D1
35 R lo loopback 65536 00:00:00:00:00:00
I have a MikroTik CRS804 switch. After connecting all 8 GB10 to the CRS804, the network stopped functioning properly.
The steps I took were:
Install OpenCode on one of GB10
Launch OpenCode and select the free DeepSeek V4 Flash model.
Connect the GB10 with OpenCdoe to the CRS804 using a USB Ethernet adapter connected to one of the CRS804’s 10G Ethernet ports.
Provide OpenCode with the CRS804 username and password (printed on the device label).
Ask DeepSeek to configure the CRS804 and the 200G network for all 8 GB10.
DeepSeek completed the entire network configuration automatically. After that, I simply asked it to generate a report summarizing the configuration, and the task was finished.
I’m using Qwen, let me try the 2-node variant and see if it does a better job. Qwen 3.5/122b has struggled here. I have found a few things out about the RouterOS help system - it’s now an F1 key based deal (for the CLI) so I can better see the valid available settings to poke and correct values for them. Still a few more to sort out though.
After watching videos and people saying ‘oh yeah just point the agent to the switch and let’er rip’ that was my expectations and it’s been far far from that easy. Maybe the model just isn’t getting it. I fed it all the community inputs/posts I could to help seed its thinking… But it really got a -lot- of things flat out wrong - like understanding it’s RouterOS and not Linux/Bash, etc. Really confused about ‘parent’ ports, etc. Somewhat AI slop tbh…
DeepSeek may be a bit better suited for this kind of task, or perhaps OpenCode’s built-in webfetch/websearch capability helped by retrieving the relevant MikroTik documentation and configuration examples. Either way, OpenCode is free to use, and the free DeepSeek tier provides a generous usage quota, so there was essentially no cost involved.
Qwen 3.5 is just underequipped for the task even if you give it the right documentation. I’ve noticed you set every lane to auto-negotiate turned off and speed to 200Gbps. That’s actually not correct. Assuming you want to split 1 400G connection into 2 200G breakouts with DAC cabling, you actually do want most of the lanes to have auto-negotiate turned on, so that the lanes you want 200G from can use them to accomplish it. If plugged into port 1 (using default names):
/interface ethernet set qsfp56-dd-1-1 auto-negotiation=no speed=200G-baseCR4
/interface ethernet set qsfp56-dd-1-2 auto-negotiation=yes
/interface ethernet set qsfp56-dd-1-3 auto-negotiation=yes
/interface ethernet set qsfp56-dd-1-4 auto-negotiation=yes
/interface ethernet set qsfp56-dd-1-5 auto-negotiation=no speed=200G-baseCR4
/interface ethernet set qsfp56-dd-1-6 auto-negotiation=yes
/interface ethernet set qsfp56-dd-1-7 auto-negotiation=yes
/interface ethernet set qsfp56-dd-1-8 auto-negotiation=yes
See - that there is exactly what I meant by having it harvest/analyze the community resources and from it found (and I think I had read myself) was claims to the contrary that you needed auto-negotiate explicitly disasbled to prevent advertising the 400g option.
This is where I wish Mikrotik or NVidia or someone had a true official ‘playbook’ for our uses given the popularity of this combo of gear.
I will say DeepSeek did a smashing job compared to Qwen. Night and day… It has worked through things and claims to have things fully configured (based on what it was given for inputs/references).
Later tonight I plan to get back to work on this home lab project, now to rearrange things where cables can actually reach where I have the Sparks setup. I’ll report back once I know more and if things are working, I’ll figure out how to share the assets and the scripts/notes surrounding them in case it saves ‘the next guy’ a headache or two. Given the community around this stuff, I’m surprised something isn’t already out there.