Save your spark cluster disk space, use this solution

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , ,
1

Hi,

This is my solution to coping the models and docker images between cluster nodes.

I moved the contents for hugging face folder of the workers nodes to another folder and deleted the entire hugging face folder from all workers. I kept only the hugging face folder on the head node.

In the head node I edited the export file to add nfs shares for every worker. Because I am not using switch it is two lines, but if you have two sparks or a switch then you will have one line. Use the IP of the connectx interfaces for speed not the WiFi or the Ethernet.

Replace your username and IP address for marked area with brackets {}

sudo nano /etc/exports

Add the following

/home/{your username}/.cache/huggingface 192.168.187.1/24(rw,sync,no_subtree_check,no_root_squash)
/home/{your username}/.cache/huggingface 192.168.177.1/24(rw,sync,no_subtree_check,no_root_squash)

Save the file then apply it

sudo exportfs -a -v

Then go to every worker node and mount the nfs share at the hugging face cache location.

sudo nano /etc/fstab

Add this line

{Connectx IP of head node}:/home/{your username}/.cache/huggingface  /home/{your username}/.cache/huggingface  nfs  defaults,_netdev,noauto,x-systemd.automount  0  0

Save the file then apply the changes

sudo mount -a

Do this for all workers nodes.

Now you don’t need to copy the model every time.

Also you can copy the unused models using normal cp command to local folders In worker nodes to have more capacity.

For docker images I use the following ssh command to copy and load the images to workers nodes using their connectx IP

First I run “docker image ls” to list all available images on head node. Then I run ssh copy like this example

docker save ghcr.io/anemll/dspark-vllm-gx10:0.1.1 | ssh -c aes128-gcm@openssh.com -o Compression=no 192.168.187.13 "docker load"

If you have better way of doing this please share it.

2

What speeds are you getting with this setup? From what I see these dram-less 2242 are quite slow with huge models files.

3

Loading speed around 7 Gigabit per second for large models. Sometimes it reaches 20 gigabit per second if the model is already loaded on ram but for very short time only.

The network is not problem. So net share is perfect for this Case.

4

yeah I was thinking of doing something similar, nfs share. Basically once the model is loaded in memory, you don’t need to use the drive for inference. For sure it will be slower the first loading, but you can save a lot of space if you only get the 1tb variant.

5

My nvme is pcie v4. So it is a little bit slower than some variants that have v5. I don’t feel this difference.

Now I have qwen 122b + qwen3.6 27 two versions + minimax m3 + glm5.2 UD q3 km + two versions of deepseek flash+ qwen 397b . In the the docker side more than 5 container variation on all nodes.

The hard drive utilization for Head nod is 50% only. And third node is at 14%. So still have plenty for experimentation

6

i would recommend connect external drive to one of sparks and share it instead of internal nvme for not very often used models. model are huge, and you can easily download couple of TB just to experiment with different quants.

7

Do you recommend any model or capacity. I have some old external SSD and hhd drives. Maybe it is the time to get new one.

8

It might be hard on the usb ports. It also a challenge to find a 2x2 enclosure that works. I have one that was reported to be working, but 10gbps is all I see.

I doubt that dram-less pcie v5 is meaningfully faster than v4 for loading huge models files.

9

New Samsung nvme reaching the limit of the bus bandwidth at 14 gigabit per second.

Maybe in the future the capacity will be higher so we can upgrade our nvme drives to higher capacity at reasonable price.

10

I looked at using a NFS export from the node(s) to share models across the cluster as well, but the 4TB NVMe drives can get a bit warm with I/O, and cooling for the NVMe isn’t that great.

I’ve since moved the huggingface cache folder off to a UGREEN DXP480T Plus with 4x4TB NVMe SSDs in RAID5 (about 11TB usable). It’s been working well; even though it’s limited to 10Gbit/s (about 1.1GB/s) when loading models, it’s actually pretty decent overall, and the drive temperatures remain very reasonable, rarely breaking 65C.
This way, I can use the local disk on the cluster nodes as a ‘playground’/scratch area, and not worry about running out of space if I decide to grab larger models on a whim.

Trying really hard not to grab a ConnectX-7 for my big NAS server now, heh :D

11

This is very good option. Although it is 7 times slower, but NAS with huge RAM give you the opportunity to implement RAG and Lmcache to expand the KV cache and load larger models.

12

Using NFS shares is a good approach for model sharing. For container sync, unless you spin up a local container registry, ssh-based save | load is probably the best working approach. (spinning up a local container registry is a lot of overhead just to facilitate file transfers, but it would enable parallel transfer of container layers; that’s basically the only alternative).

You can use sparkrun to automate and coordinate running inference models including container distribution (it checks for container drift and copies as needed). It also supports you configuring a NFS cache directory and then it’ll handle distribution / not distributing as needed.

sparkrun is brought to you by the Spark Arena team (myself, @raphael.amorim, @eugr). You can also view and publish benchmarks at https://spark-arena.com

13

Thank you for the great work at spark run project. We all appreciate the effort done. I didn’t know it have all those features. I am using spark-vllm version.

I had small problem with cluster initializing for 3 spark mesh last month using sparkrun, but it was nccl problem, so I kept what i have. I am waiting for Version 3 of spark run. Hopefully it will be released soon.

14

Storage management is one of the first things that breaks on a Spark cluster once people start pulling multiple 40–80GB models per node. In addition to the tool you shared, a few patterns that scale better:

- Shared NFS/CEPH model cache across nodes instead of local Hugging Face caches. Mount /models read-only and let vLLM/TensorRT-LLM load from there. Eliminates duplication.

- Reference-counted model layers if you run heterogeneous workloads. Many models share the same base tokenizer/architecture; store one copy and symlink.

- TTL on generated artifacts: set a cron or systemd timer to delete checkpoints/outputs older than N days unless tagged.

- Docker image hygiene: docker system prune -a --volumes with caution, and prefer base images that layer cleanly so rebuilds don’t bloat /var/lib/docker.

Before anyone runs a big cleanup, snapshot the model cache list. Redownloading 200GB because a script was too aggressive is a painful lesson.

15

I wish all of this house keeping is integrated to one of the tools like sparkrun so it could be done with a one click.

16

sparkrun has support for that already - i pull and store on nas with it