I’m running two DGX Sparks as a small cluster (ConnectX-7 link between them, nightly borg backups) and I’m about to buy external storage for a proper backup tier plus a model library. Before I do, I’d like to hear what other Spark owners are actually running day-to-day.
My planned setup:
- TerraMaster D8 Hybrid over USB-C (USB 3.2 Gen2 10Gbps): 4x 3.5" SATA bays + 4x M.2 NVMe slots. Bays 1-2 in hardware RAID 1 with 2x IronWolf Pro 8TB as the borg target; one 4TB NVMe as a model library (GGUF/safetensors, vector DB snapshots); spare bays for a raw market-data mirror
- Model library NFS-exported over the ConnectX-7 link so both Sparks share one store (same mount path + container bind-mounts on both nodes) instead of copying models to each node
- Second Spark holds a lagged rsync replica of the borg repos over the same link
- Small encrypted cloud repo for the irreplaceable subset
Questions for people with real hours on external storage:
-
USB enclosure compatibility: anyone running a TerraMaster D8 Hybrid (or the newer D8 Hybrid 2 revision, or siblings like the D5 Hybrid / D4-320) on a Spark? Specifically: does the hardware RAID 1 volume present cleanly to DGX OS and survive reboots/power events without the array needing re-recognition? Do the M.2 slots and SATA bays both behave under UAS, and does SMART pass through (smartctl -d sat) for the drives behind the bridge? Any disconnects or link renegotiation under sustained transfers? If you tried a different multi-bay SATA box (QNAP TR-004, OWC, Sabrent) and it worked better or worse on the Spark, that’s exactly what I’m after.
-
24/7 stability: anyone running a USB DAS continuously attached for months? The Spark’s USB power/sleep behavior has some known oddities (e.g. the fans-stop-on-headless issue) — anything similar affecting attached storage?
-
Hardware RAID vs mdadm: on DGX OS, are people trusting enclosure hardware RAID, or presenting individual disks and using mdadm? Any SMART passthrough issues with specific bridges?
-
Model library placement: for those serving models from external NVMe (~1-2 GB/s over USB) vs internal — is external load speed acceptable in practice, or did you end up shuffling everything to the internal M.2 anyway? (I’ve seen reports that ASM2464-based NVMe enclosures behave well on the Spark’s 20Gbps ports while JHL7440-based ones underperform — confirmations welcome.)
-
Alternatives I should consider: anyone doing this over the network instead (NAS via the 10GbE port, or storage hung off the ConnectX-7 link) and glad they did?
-
Shared model store over the cluster link: anyone NFS-exporting a model library from one Spark and mounting it on the other (nconnect, or NFS-over-RDMA on the CX7) so containers on both nodes see one store? Any DGX OS wrinkles running nfs-kernel-server, or with mmap’d model loads over NFS? Most setups I’ve seen copy models to every node and I’d like to avoid that.
-
NVMe-oF on the QSFP ports: has anyone exported NVMe-oF between Sparks (kernel nvmet target over RoCE on the CX7 link), or from a separate target box into a Spark’s spare QSFP port? Curious whether DGX OS ships the initiator/target modules ready to go and what real throughput looks like.
Not looking for general backup philosophy — specifically interested in hardware combos verified working (or failing) on a Spark. I’ll report back with whatever I end up building.