cudaMemcpyAsync (P2P D2D) serializes with kernel execution

[abhishekghos]

Persistent Kernel does not work properly on some GPUs

This appears to be related to CUDA lazy module loading. The topic is covered in a few places, including here and here. Note that the changes were initially introduced in the CUDA 11.7 timeframe as an opt-in and then became default in the CUDA 12.2 timeframe. So one possible source of differences in observations may be the version of CUDA in each case. Briefly, for the purpose of this discussion, a module can be thought of as the code for a kernel. In some cases, loading that kernel code may…

Seems like lazy loading causes the issue, and the following fixes it:

CUDA_MODULE_LOADING=EAGER ./my_app

Will reason about the answer properly, and write it down…