Nsight Compute multi-node profiling hangs at "Waiting for TCP clients to attach" when using --communicator tcp

Developer Tools Nsight Compute
1

Hello,

I am trying to use Nsight Compute to profile a multi-node NCCL/PyTorch distributed application.

Single-node multi-process profiling works correctly, but multi-node profiling hangs during TCP communicator initialization.

Environment:

  • Nsight Compute: 2026.1.1.0

  • CUDA driver: 13.0

  • NCCL: 2.20.5

  • OpenMPI

The following command works correctly WITHOUT communicator options:

timeout 30s mpirun --tag-output -np 2 \
  -H 192.168.35.170,192.168.205.180 \
  -x PATH -x LD_LIBRARY_PATH \
  -x MASTER_ADDR=192.168.35.170 \
  -x MASTER_PORT=12355 \
  -x NCCL_DEBUG=INFO \
  bash -lc '
    export RANK=$OMPI_COMM_WORLD_RANK
    export WORLD_SIZE=$OMPI_COMM_WORLD_SIZE
    export LOCAL_RANK=$OMPI_COMM_WORLD_LOCAL_RANK

    ./run_ncu.sh \
      -f -o report_rank%q{OMPI_COMM_WORLD_RANK} \
      python3 ddp_ncu_test.py
  '

NCCL initializes correctly across nodes and NET/Socket communication works normally.

However, when I enable:

--communicator tcp
--communicator-tcp-hostname=192.168.35.170
--communicator-tcp-num-peers=2
--lockstep-kernel-launch

the application hangs during communicator initialization:

==PROF== Waiting for TCP clients to attach on 192.168.35.170:49217
==PROF== Waiting for TCP clients to attach on 192.168.205.180:49217

and no further progress occurs.

The command is:

timeout 30s mpirun --tag-output -np 2 \
  -H 192.168.35.170,192.168.205.180 \
  -x PATH -x LD_LIBRARY_PATH \
  -x MASTER_ADDR=192.168.35.170 \
  -x MASTER_PORT=12355 \
  -x NCCL_DEBUG=INFO \
  bash -lc '
    export RANK=$OMPI_COMM_WORLD_RANK
    export WORLD_SIZE=$OMPI_COMM_WORLD_SIZE
    export LOCAL_RANK=$OMPI_COMM_WORLD_LOCAL_RANK

    ./run_ncu.sh \
      --communicator tcp \
      --communicator-tcp-hostname=192.168.35.170 \
      --communicator-tcp-num-peers=2 \
      --lockstep-kernel-launch \
      -f -o report_rank%q{OMPI_COMM_WORLD_RANK} \
      python3 ddp_ncu_test.py
  '

Important observations:

  1. Single-node multi-process profiling works correctly with communicator enabled.

  2. Multi-node NCCL communication works correctly without communicator enabled.

  3. TCP connectivity between nodes is verified.

  4. Using hostname instead of IP does not help:
    –communicator-tcp-hostname=powerleader

  5. Removing --communicator-tcp-port and using dynamic ports does not help either.

  6. strace analysis shows:

    • all peers call bind()/listen()/accept()

    • but no peer ever calls connect()

    • “Initialized Communicator” never appears

Example strace behavior:

bind(8<TCP:[...]>, {AF_INET ...})
listen(8<TCP:[0.0.0.0:49217]>, 128)
accept(...) = -1 EAGAIN
epoll_wait(...)

but no:

connect(...)

Questions:

  1. Does Nsight Compute communicator internally require hostname reverse resolution between nodes?

  2. Does communicator automatically select network interfaces internally?

  3. Is there any way to specify communicator bind interface/IP address?

  4. Is multi-node communicator expected to work with MPI + NCCL + lockstep kernel launch?

  5. Why do all peers appear to enter listen mode without any TCP connect() occurring?

Any guidance would be greatly appreciated.

Thank you.