I have three spark in full mesh/ring configuration as per spark-vllm repo netplan. When I installed sparkrun run it doesn’t recognize the cluster correctly and trying to do everything again, but fail and mixed up the netplan. Now I am back using spark-vllm, but can’t enjoy new recipes from sparkrun.
I think if anyone would share with me his cluster files from .config/sparkrun/cluster then I can figure out what I am missing.
Note: that I am using this configuration to run deepseek 4 flash with nccl. But I don’t need it for other models.
recipe_version: "1" name: deepseek-v4-flash description: "DeepSeek-V4-Flash" container: aidendle94/sparkrun-vllm-ds4-gb10:production-v2 mods: [] defaults: model: /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash/snapshots/bc486f653513c9179e20a970587dcbe928bf7b96 port: 8000 host: 0.0.0.0 tensor_parallel: 2 pipeline_parallel: 1
data_parallel: 1
env:
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_TRITON_MLA_SPARSE: "1"
VLLM_TRITON_MLA_SPARSE_HEAD_BLOCK_SIZE: "4"
VLLM_TRITON_MLA_SPARSE_QUERY_CHUNK_SIZE: "512"
VLLM_TRITON_MLA_SPARSE_TOPK_CHUNK_SIZE: "512"
HF_HUB_OFFLINE: "1"
# === NCCL: built-in IB transport (not plugin) ===
NCCL_NET: "IB"
NCCL_DEBUG: "WARN"
command: |
# Per-node HCA selection using VLLM_HOST_IP (set by launch-cluster.sh)
case "$VLLM_HOST_IP" in
192.168.100.140) export NCCL_IB_HCA="rocep1s0f1" ;;
192.168.100.138) export NCCL_IB_HCA="rocep1s0f0" ;;
192.168.100.162) export NCCL_IB_HCA="rocep1s0f0" ;;
esac
echo "Node: $VLLM_HOST_IP -> NCCL_IB_HCA=$NCCL_IB_HCA"
vllm serv....
The problem that it assume the wrong IP address for cx7 hosts
Hosts: default cluster 'mesh'
Head: 192.168.100.138
Workers: 192.168.100.140
[1/6] Preparing
done (0.0s)
[2/6] Building image — skipped (no builder)
[3/6] Distributing resources
SSH cmd <- 192.168.198.12 FAILED rc=255 (5.0s): ssh: connect to host 192.168.198.12 port 22: Connection timed out
SSH cmd <- 192.168.197.12 FAILED rc=255 (5.0s): ssh: connect to host 192.168.197.12 port 22: Connection timed out
Distributing image aidendle94/sparkrun-vllm-ds4-gb10:production-v2 to 2 host(s)
Container image up-to-date on all 2 host(s)
Distributing model unsloth/DeepSeek-V4-Flash to 2 host(s)
Fetching 74 files: 100%|████████████████████████████████████████████████████████| 74/74 [00:00<00:00, 10179.35it/s]
Download complete: : 0.00B [00:00, ?B/s] Model synced to 2 host(s) | 0/74 [00:00<?, ?it/s]
done (8.2s)
[4/6] Syncing tuning configs
done (0.0s)
[5/6] Launching vllm runtime
Step 1/7: Cleaning up existing containers
Step 2/7: Detecting InfiniBand
Step 3/7: Detecting head node IP
Step 4/7: Launching containers
Step 5/7: Running pre-serve hooks
Step 6/7: Starting head node serve
SSH script <- 192.168.100.138 FAILED rc=1 (5.6s): ============================================================
ERROR: Serve process exited immediately (PID 57)
Container logs:
============================================================
Failed to exec serve on head node: ============================================================
ERROR: Serve process exited immediately (PID 57)
Container logs:
============================================================
This is from the other node
Available for KV: 99.8 GB
Warning: model_params not available; model weight estimate is zero
Warning: Missing architecture info (num_layers, num_kv_heads, head_dim); KV cache estimate unavailable
Hosts: default cluster 'mesh'
Head: 192.168.100.140
Workers: 192.168.100.138
[1/6] Preparing done (0.0s) [2/6] Building image done (0.0s) [3/6] Distributing resources SSH cmd <- 192.168.188.11 FAILED rc=255 (5.0s): ssh: connect to host 192.168.188.11 port 22: Connection timed out SSH cmd <- 192.168.187.11 FAILED rc=255 (5.0s): ssh: connect to host 192.168.187.11 port 22: Connection timed out Distributing image aidendle94/sparkrun-vllm-ds4-gb10:production-v2 to 2 host(s) Container image up-to-date on all 2 host(s) done (6.4s) [4/6] Syncing tuning configs done (0.0s) [5/6] Launching vllm runtime Step 1/7: Cleaning up existing containers Step 2/7: Detecting InfiniBand Step 3/7: Detecting head node IP Step 4/7: Launching containers Step 5/7: Running pre-serve hooks Step 6/7: Starting head node serve SSH script <- 192.168.100.140 FAILED rc=1 (5.6s): ============================================================ ERROR: Serve process exited immediately (PID 17) Container logs: ============================================================ Failed to exec serve on head node: ============================================================ ERROR: Serve process exited immediately (PID 17) Container logs: ============================================================