GLM-4.7-FP8 on 4× DGX Spark via SGLang: 2.5× speedup (8.2 → 25 tok/s) just by enabling RDMA

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

I spent most of a day tuning GLM-4.7-FP8 on a 4× DGX Spark cluster and stalled at 9.8 tok/s decode after CUDA graphs were enabled. The published recipe (forum thread 359256) claims 20–27 tok/s. Same hardware class, but I was burning a chunk of the budget on NCCL socket transport without realizing it.

The fix is one line: enable RDMA on the SGLang container.

After flipping, single-stream decode jumped from 9.8 → 25.1 tok/s (2.56×) on the same model, same flags, same EAGLE config. Verified RDMA was actually engaged by checking via NET/IB lines in NCCL_DEBUG=INFO output.

Posting this because the playbooks I followed for SGLang don’t call out the device passthrough explicitly — the vLLM venv stack inherits /dev/infiniband from the host automatically, but SGLang in Docker does not. Easy to miss.

Hardware

  • 4× DGX Spark (GB10, sm_121a, 128 GiB UMA per node)
  • ConnectX-7 RoCE NICs (visible as rocep1s0f0 via ibv_devices)
  • 200 GbE switched (MikroTik), MTU 1500
  • vLLM 0.22.1 venv working at full RDMA already — SGLang container was the laggard

What I changed in the SGLang launch (the only changes)

Env block (added to docker run)

-e NCCL_IB_HCA=rocep1s0f0
-e NCCL_IB_DISABLE=0          # was 1

(Remove NCCL_P2P_DISABLE=1 if you have it set — not needed once IB is up.)

Docker run args (added)

--device=/dev/infiniband
--cap-add=IPC_LOCK
--ulimit memlock=-1

Verification (after launch, before serving probes)

ssh head-node 'docker logs sglang-server 2>&1 | grep -c "via NET/IB"'   # want N > 0
ssh head-node 'docker logs sglang-server 2>&1 | grep -c "via NET/Socket"'  # want 0

If you see NET/IB : No device found, the device passthrough didn’t take — re-check --device=/dev/infiniband and that the container has libibverbs.so (ldconfig -p | grep ibverbs inside).

Throughput (single user, P1 reasoning probe, 1100-token output)

Config Single-stream n=4 agg n=8 agg
Socket NCCL (default in most SGLang containers) 8.2 — —
+ Full CUDA graphs 9.8 — —
+ RDMA enable (this post) 25.1 56 79

Why this works on Spark

The Spark ConnectX-7 NICs expose /dev/infiniband/{rdma_cm,umad0,umad1} at the host. ibv_devices shows rocep1s0f{0,1} and roceP2p1s0f{0,1} — these are RoCE devices, but they register with the IB userspace API. Once the container gets the device + IPC_LOCK + libibverbs, NCCL picks them automatically.

Pattern likely applies to any SGLang container on 4× Spark — same fix probably gives a 2–3× boost on Nemotron-3-Ultra, MiniMax variants, and any other MoE multi-node setup that’s currently on socket transport.

What I’m running

  • Container: SGLang built from BTankut’s source (similar to lmsysorg/sglang:spark shape)
  • Model: zai-org/GLM-4.7-FP8, 4× Spark TP=4, EAGLE on every node, CUDA graphs ON, full FlashInfer attention
  • Cluster: spark-1/2/3/4 over MikroTik 200 GbE
  • Reference recipe I started from: thread 359256

Happy to share the full launch script if useful. Just call out that the RDMA delta itself is the headline finding — everything else in the recipe is roughly the published GLM-4.7 4× Spark config.

1 Like