When replaying an CUDA graph, what should be done if there is some interaction with the CPU?
For example, in NCCL, the CPU needs to perform RDMA operations to write to the peer GPU’s memory, and after flushing, the GPU kernel can then access the data for further reduce and copy operations.
Since replaying does not involve the CPU, how can the RDMA operations and kernel operations be coordinated to depend on each other when replaying?
Yes, capturing arbitrary library calls in a CUDA graph may have unexpected results since ordinary host-side work is not replayed.
For NCCL, CUDA graph is explicitly supported for NCCL’s collective, P2P and group operations`, see Using NCCL with CUDA Graphs — NCCL 2.27.5 documentation .
Whether this involves RDMA operations, or how they are handled internally, I do not know.
The standard CUDA way to support the capture of non-CUDA-API CPU work is to launch the host-side work via cudaLaunchHostFunc