NCCL socket transport fails with pipeline parallelism (mesh_pp) on DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , ,
1

Hey folks,
I’m running into a persistent NCCL error with pipeline parallelism on two DGX Spark nodes and I’m stuck. I tried to keep it short, because I respect your time and appreciate you reading this post. The first NCCL communicator initializes fine, but the second one (for pipeline parallelism) fails with what looks like a socket protocol mismatch. I’m not entirely sure this the correct approach, but my end goal is to fine-tune a 70B model. My current goal is to test if it’s possible, and then get some benchmarks completed.

Environment:

  • Hardware: 2x NVIDIA DGX Spark (GB10 Grace Blackwell)
  • OS: DGX OS (Linux 6.14.0-1015-nvidia)
  • PyTorch: 2.10.0a0+b558c986e8.nv25.11 CUDA 13.0
  • NCCL: 2.28.8+cuda13.0
  • NeMo AutoModel: 0.2.0
  • Network: Direct DAC connection (ConnectX-7, 192.168.100.10 and .11)

Configuration:

model: meta-llama/Llama-3.3-70B-Instruct
quantization: 8-bit (BitsAndBytes)
peft: LoRA (rank=16, alpha=32)

distributed:
pp_size: 2
dp_size: none
tp_size: 1

autopipeline:
pp_schedule: interleaved1f1b
pp_microbatch_size: 1
layers_per_stage: 20

The error:

The benchmark fails at iteration 0 with:

[2026-01-01 21:57:19] dgx1:1007:1165 [0] transport/net_socket.cc:565 NCCL WARN NET/Socket :
peer 192.168.100.11<56630> message truncated : receiving 8 bytes instead of 4.
If you believe your socket network is in a healthy state, there may be a mismatch in
collective sizes or environment settings (e.g. NCCL_PROTO, NCCL_ALGO) between ranks

[rank0]:[E101 21:57:19.147805485 ProcessGroupNCCL.cpp:617] [Rank 0] Collective
WorkNCCL(SeqNum=1, OpType=COALESCED, NumelIn=0, NumelOut=0, Timeout(ms)=600000)
raised the following async exception: NCCL error: invalid usage

[rank0]:[E101 21:57:19.150803196 ProcessGroupNCCL.cpp:2290] [PG ID 1 PG GUID 1(mesh_pp)

Rank 0] failure detected by watchdog at work sequence id: 1

What works:

The first NCCL communicator (default_pg) initializes perfectly:

dgx1:1007:1007 [0] NCCL INFO ncclCommInitRankConfig comm 0x389afbf0 rank 0 nranks 2 - Init COMPLETE

dgx1:1007:1007 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0

Gloo backend connects successfully. Model loads and splits correctly across pipeline stages.

What fails:

The second NCCL communicator (mesh_pp) for pipeline parallelism. It reports “Init COMPLETE” but crashes immediately on the first collective operation with “message truncated: receiving 8 bytes instead of 4.”

What I’ve tried:

Network configuration:

  • Verified single IP per node (192.168.100.10 and .11)
  • Confirmed MTU=1500 on both nodes
  • Tested ping connectivity (< 1.5ms latency)
  • Removed secondary DAC interface IPs to prevent subnet conflicts
  • Set NCCL_SOCKET_IFNAME=enp1s0f0np0 on both nodes

NCCL environment variables:

export NCCL_DEBUG=INFO
export NCCL_NET_PLUGIN=none
export NCCL_SOCKET_IFNAME=enp1s0f0np0
export NCCL_ALGO=Ring
export NCCL_PROTO=Simple

**I also tried NCCL_IB_DISABLE=1 (causes double-free crash in RDMA plugin), various buffer size settings (no effect).

Process cleanup:

  • Verified ports are free
  • No lingering Python processes
  • Flushed page cache before benchmark

Additional troubleshooting steps:

  • Multiple GPU resets after crashes to clear stuck states
  • Attempted to enable RoCE/IB transport with NCCL_IB_DISABLE=1 (causes double-free crash in RDMA plugin, forcing fallback to socket transport)
  • Tested multiple NCCL protocol/algorithm combinations individually and together
  • Explicitly verified in NCCL debug logs that both ranks show identical NCCL_ALGO=Ring and NCCL_PROTO=Simple settings
  • Confirmed both nodes report the same settings during initialization, yet the error persists

Observations:

The failure seems completely deterministic - same point every time (iteration 0, first collective on mesh_pp). The “receiving 8 bytes instead of 4” suggests incompatible message formats during handshake. It only seems to affect mesh_pp process group, not default_pg.

The error message suggests checking for NCCL_PROTO and NCCL_ALGO mismatches between ranks, but I’ve verified both ranks have identical settings (from NCCL debug logs). The error happens even when both ranks explicitly use NCCL_ALGO=Ring and NCCL_PROTO=Simple.

Questions:

Is this a known limitation of NCCL socket transport with multiple process groups? Or am I missing something for pipeline parallelism over socket transport? The first communicator works perfectly with the same network and NCCL settings, so I’m confused why the second one fails during handshake. I started off with tensor parallelism but ran into issues I couldn’t resolve; should I be using tensor parallelism instead?

I appreciate you and your time. Any help would be appreciated. Thanks!

2

I realized I was using November’s image, so I upgraded to PyTorch 2.10.0a0+b4e4ee81d3.nv25.12 with NCCL 2.28.9+cuda13.1 to see if newer NCCL version would resolve the issue. Unfortunately, the problem persists but with a different error pattern.

Previous environment:

  • PyTorch: 2.10.0a0+b558c986e8.nv25.11
  • NCCL: 2.28.8+cuda13.0
  • Error: “message truncated: receiving 8 bytes instead of 4”

New environment:

  • PyTorch: 2.10.0a0+b4e4ee81d3.nv25.12
  • NCCL: 2.28.9+cuda13.1
  • Error: “unhandled system error” during communicator creation

New error pattern:

The mesh_pp communicator now fails during creation with repeated timeout errors:

[Rank 1] Call NCCL collective ncclAllGather_v2 failed: unhandled system error
[Rank 1] NCCL WARN [Proxy Service] Error: handler returned remote error (ncclRemoteError/-5), source proxy progress recv thread

The communicator enters a retry loop but never successfully initializes. Same NCCL environment variables (NCCL_NET_PLUGIN=none, NCCL_SOCKET_IFNAME=enp1s0f0np0, NCCL_ALGO=Ring, NCCL_PROTO=Simple) applied on both ranks.

At this point, I think I’ve gotta be doing something wrong. I started off by following one of the tutorials but feel like I’ve gotten a bit too far into the weeds, here.

3

Have you first tried our NCCL playbook? Want to make sure it’s an application issue and not something deeper.

4

Hey Aniculescu, thanks for the response!

I did happen to follow the NCCL tutorial, which was super helpful getting started. After reading your comment, I went back and read through the tutorial again. I didn’t see anything that was missed, or later replaced, so I ran a 32GB test to make sure it’s still working as intended. I got the bandwidth I was expecting and didn’t run into any errors (beyond the authorization required nag).

Authorization required, but no authorization protocol specified
nccl-tests version 2.17.7 nccl-headers=22809 nccl-library=22809
Collective test starting: all_gather_perf
nThread 1 nGpus 1 minBytes 34359738368 maxBytes 34359738368 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0

Using devices
Rank  0 Group  0 Pid   6504 on       dgx1 device  0 [000f:01:00] NVIDIA GB10
Rank  1 Group  0 Pid   3914 on       dgx2 device  0 [000f:01:00] NVIDIA GB10

                                                         out-of-place                       in-place          
  size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong 
   (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)         

34359738368    4294967296     float    none      -1   675833   50.84   25.42       0   670125   51.27   25.64       0
Out of bounds values : 0 OK
Avg bus bandwidth    : 25.5286

Collective test concluded: all_gather_perf

I’ve been assuming it’s user error, because I imagine someone else would’ve ran into this already. It would be neat if I found a bug, though; let me know if there’s more information I can provide or troubleshooting that you recommend I perform. I’m a bit out of ideas, so I was thinking about trying vLLM out in a new container. I’m still pretty new and am testing the different waters.

Thanks again!

5

Do you need PROTO and ALGO set?
For RDMA, you need to specify IB devices in NCCL_IB_HCA, too, otherwise it will always fall back to socket.
You can also set a few extra variables just in case:

export UCX_NET_DEVICES=enp1s0f1np1
export NCCL_SOCKET_IFNAME=enp1s0f1np1
export OMPI_MCA_btl_tcp_if_include=enp1s0f1np1
export NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1
export NCCL_IB_DISABLE=0
export GLOO_SOCKET_IFNAME=enp1s0f1np1
export NCCL_DEBUG=INFO

and enable debug for more

6

Thanks for the recommendations, that really helped!!

RDMA is working, now. The missing piece was --device=/dev/infiniband/ on the Docker containers. Once I added that along with your recommended env vars NCCL found the IB devices and I can see it opening /dev/infiniband/uverbs0 and uverbs2 in the process file descriptors.

I also had to remove some secondary IPs (192.168.100.14/.15) on the second ConnectX-7 ports because NCCL was preferring those interfaces over the primary ones even with NCCL_SOCKET_IFNAME set. Weird.

Now I’m running into something else, and it’s really making me wonder if there’s a better way to do this and if I took a wrong turn somewhere…

KeyError: "Invalid mesh_dim_names ('dp_replicate', 'dp_shard_cp', 'tp') specified. 
Mesh dim indices should be in ascending order."

Error occurs at parallelizer.py:973 during fsdp2_strategy_parallelize. I’ve tried:

  • TP=1, PP=2 with simplified 1f1b schedule (4-bit quantization)
  • TP=1, PP=2 with interleaved1f1b (4-bit quantization)
  • PP=2 with interleaved1f1b (8-bit quantization)
  • PP=2 with simplified 1f1b schedule (8-bit quantization)
  • TP=2 (no quantization, lolYOLO)

All fail with the same mesh ordering error. Environment is PyTorch 25.11 container (2.10.0a0+nv25.11, CUDA 13.0), NeMo AutoModel 0.2.0.

Current config looks like:

distributed:
  _target_: nemo_automodel.components.distributed.fsdp2.FSDP2Manager
  dp_size: none
  tp_size: 1
  cp_size: 1
  pp_size: 2

peft:
  _target_: nemo_automodel.components._peft.lora.PeftConfig
  match_all_linear: True
  dim: 16

Any ideas? I’m wondering if I should keep troubleshooting or start looking at a new approach. I’ve really only gotten this far by chasing rabbits.

Also, no, I don’t think I needed PROTO and ALGO set. I was just following a suggestion from an error message, haha.

I appreciate you and your time!