Deploying DeepSeek V4 Flash 0731 on Dual DGX Spark with RoCE: A Complete Guide

AI & Data Science NVIDIA Nemotron
1

Deploying DeepSeek V4 Flash 0731 on Dual DGX Spark with RoCE: A Complete Guide

This post shares my hands-on experience deploying the DeepSeek V4 Flash 0731 (156GB, ~685B MoE) model across two NVIDIA DGX Spark workstations interconnected via RoCE (RDMA over Converged Ethernet).

Hardware Topology

┌─────────────────────┐          ┌─────────────────────┐
│   DGX Spark (Head)  │  RoCE    │  DGX Spark (Worker) │
│  IP: 192.168.22.160 │◄────────►│  IP: 192.168.22.161 │
│  Node Rank: 0       │  RDMA    │  Node Rank: 1       │
└─────────────────────┘          └─────────────────────┘
       GPUs: 2x GB10                    GPUs: 2x GB10
       Memory: 288GB                    Memory: 288GB

Key Configuration Parameters

1. RoCE / NCCL Configuration

Proper RDMA configuration is critical for dual-machine deployment. Key environment variables:

export NCCL_IB_GID_INDEX=3
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_QPS_PER_CONNECTION=4
export NCCL_SOCKET_IFNAME=enp194s0f0np0
export NCCL_IB_HCA=mlx5_0,mlx5_1

2. Model Serving Parameters

--max-model-len 131072
--gpu-memory-utilization 0.78
--max-num-seqs 6

Native 1M Context: YaRN with factor=16 for native 1M context. --max-model-len controls the serving window.

Memory Safety: 0.78 is safe ceiling (~79GB/GPU of 288GB). Higher risks OOM under load.

Concurrency: 6 gives good balance. Adjustable 4-12.

Deployment Steps

On Head Node (192.168.22.160):

docker run -d --name vllm_head \
  --network host --privileged --ipc=host --gpus all \
  -v /path/to/model:/model \
  --env NODE_RANK=0 --env VLLM_HOST_IP=192.168.22.160 \
  vllm_anemll:latest \
  --model /model --max-model-len 131072 \
  --gpu-memory-utilization 0.78 --max-num-seqs 6

On Worker Node (192.168.22.161):

docker run -d --name vllm_worker \
  --network host --privileged --ipc=host --gpus all \
  -v /path/to/model:/model \
  --env NODE_RANK=1 --env VLLM_HOST_IP=192.168.22.161 \
  vllm_anemll:latest \
  --model /model --max-model-len 131072 \
  --gpu-memory-utilization 0.78 --max-num-seqs 6 \
  --headless

Critical Notes

  • Worker needs --headless; Head must NOT have it
  • --network host is required (NCCL RoCE RDMA needs direct NIC)
  • Start Worker first, then Head
  • NCCL init: ~2 min | Model loading: ~150 sec per node
  • Full startup: ~5-7 minutes

Performance

  • TTFT: ~120ms
  • Throughput: ~15-20 tok/s per request
  • GPU Utilization: ~96% during inference
  • Memory: ~79GB per GPU

Lessons Learned

  1. NODE_RANK unique: Worker=1, Head=0. Conflicts break NCCL.
  2. Start order: Worker first, Head after.
  3. No NAT: Bridge/NAT breaks RDMA.
  4. RoCE GID: Auto-detect or set explicitly.
  5. Memory ceiling: 0.78 safe max.

Source Code

Full scripts and docs: alexlu0912/dgxspark_deepseekv4flash0731

Happy to answer questions!