Don't observe overlapping behavior in streams

I’m trying to view overlapping behavior in kernel launches and memcpy for streams on my L40S gpu server running Ubuntu 22.04. For some reason, I fail to see any overlap in any of the memcpy/kernels:


Both the kernels and the kernel launches end up being sequentially ordered. My memcpy is also rather small (4Kb).

Some details of my device properties

can map host memory to cuda addresses : Yes
max threads per block : 1024
concurrent kernels supported : Yes
Overlap allowed in kernel and memcopy : Yes
Compute mode : 0
Async engines : 2

The code is compiled with --default-stream per-thread for sm_89

#include <cuda.h>
#include <cuda_runtime.h>

#include <cassert>
#include <cstdio>

__global__ void computeSquares(int *deviceArray, int *resultArray);
__global__ void computeCubes(int *deviceArray, int *resultArray);

int main() {
  // Allocate page locked memory on the host
  const int ArraySize = 1024;
  const int BlockSize = 128;
  const int GridSize = ArraySize / BlockSize;
  const int ArraySizeInBytes = ArraySize * sizeof(int);

  int *hostArray;
  cudaMallocHost(&hostArray, ArraySizeInBytes);

  // Allocate memory for the results
  int *squares, *cubes;
  cudaMallocHost(&squares, ArraySizeInBytes);
  cudaMallocHost(&cubes, ArraySizeInBytes);

  // Initialize array elements to 10
  for (int i = 0; i < ArraySize; i++) {
    hostArray[i] = 10;
  }

  // Create a stream for each operation
  cudaStream_t squareStream, cubeStream;
  cudaStreamCreateWithFlags(&squareStream, cudaStreamNonBlocking);
  cudaStreamCreateWithFlags(&cubeStream, cudaStreamNonBlocking);

  // Copy the array to the device
  int *inputArrSquares;
  cudaMallocAsync(&inputArrSquares, ArraySizeInBytes, squareStream);
  cudaMemcpyAsync(inputArrSquares, hostArray, ArraySizeInBytes,
                  cudaMemcpyHostToDevice, squareStream);
  int *inputArrCubes;
  cudaMallocAsync(&inputArrCubes, ArraySizeInBytes, cubeStream);
  cudaMemcpyAsync(inputArrCubes, hostArray, ArraySizeInBytes,
                  cudaMemcpyHostToDevice, cubeStream);

  // Allocate memory on the device to store the results
  int *devSquaredArray;
  cudaMallocAsync(&devSquaredArray, ArraySizeInBytes, squareStream);

  int *devCubedArray;
  cudaMallocAsync(&devCubedArray, ArraySizeInBytes, cubeStream);

  // Launch kernel
  computeSquares<<<GridSize, BlockSize, 0, squareStream>>>(inputArrSquares,
                                                           devSquaredArray);
  computeCubes<<<GridSize, BlockSize, 0, cubeStream>>>(inputArrCubes,
                                                       devCubedArray);

  // Copy the result back to the host
  cudaMemcpyAsync(squares, devSquaredArray, ArraySizeInBytes,
                  cudaMemcpyDeviceToHost, squareStream);

  cudaMemcpyAsync(cubes, devCubedArray, ArraySizeInBytes,
                  cudaMemcpyDeviceToHost, cubeStream);

  // Synchronize ALL work on the device
  cudaDeviceSynchronize();

  // Sanity check
  for (int i = 0; i < ArraySize; i++) {
    assert(squares[i] == 100);
    assert(cubes[i] == 1000);
  }
}

__global__ void computeSquares(int *deviceArray, int *resultArray) {
  int index = threadIdx.x + blockIdx.x * blockDim.x;
  resultArray[index] = deviceArray[index] * deviceArray[index];
}

__global__ void computeCubes(int *deviceArray, int *resultArray) {
  int index = threadIdx.x + blockIdx.x * blockDim.x;
  resultArray[index] =
      deviceArray[index] * deviceArray[index] * deviceArray[index];
}

I was initially copying the host array to a location in memory that both kernels read from. I thought that since the memory was not marked read only, the scheduler ended up serializing the kernels. I modified my code to use separate device memory for each kernel (I don’t know how to mark device memory as read only). However, this made no difference to the observed behavior of the streams.

Any clues on why I don’t see any overlapping behavior?

Your kernels are on the order of 5-10 us in duration (!). That is very short to try to aim for a particular concurrency scenario. I would call your work-issuance strategy a “breadth-first” issuance order. That is you do all the H->D copy issuance, followed by the kernel issuance, followed by the kernels, followed by D->H. To witness overlap with a very short kernel, I would probably also try “depth first” issuance.

Additionally, while cudaMallocAsync is designed to be a lighter weight allocator, I would still encourage folks to get unnecessary steps out of the work issuance sequence. Expecting zero impact to asynchronous work delivery any and every time you issue cudaMallocAsync might not be realistic. The first time you use such an allocator may incur additional overhead, and there may be subsequent points where additional overhead occurs. It’s not a free lunch in my experience.

Finally, in modern CUDA settings, a kernel, the first time it is launched, is a synchronizing operation with respect to device activity (due to so-called “lazy initialization”). This can be troublesome when trying to witness overlap in a toy scenario, or the first (and only, in this case) time you launch something.

The first thing I would do if I wanted to witness copy/compute overlap would be to increase the duration of the kernel, probably to at least 50 us. I also believe it is necessary to fix the lazy initialization issue. You can play with the other ideas to see if they matter (depth-first, cudaMallocAsync out of the work issuance area).

This variation of your code on my L4 seems to show some overlap kernel-kernel and kernel-memcpy:

# cat t395.cu
#include <cuda.h>
#include <cuda_runtime.h>

#include <cassert>
#include <cstdio>
const int lc = 10000;
__global__ void computeSquares(int *deviceArray, int *resultArray, bool quickExit = false) {
  if (quickExit) return;
  int index = threadIdx.x + blockIdx.x * blockDim.x;
  for (int i = 0; i < lc; i++) resultArray[index] = deviceArray[index] * deviceArray[index];
}

__global__ void computeCubes(int *deviceArray, int *resultArray, bool quickExit = false) {
  if (quickExit) return;
  int index = threadIdx.x + blockIdx.x * blockDim.x;
  for (int i = 0; i < lc; i++) resultArray[index] =
      deviceArray[index] * deviceArray[index] * deviceArray[index];
}
int main() {
  // Allocate page locked memory on the host
  const int ArraySize = 1024;
  const int BlockSize = 128;
  const int GridSize = ArraySize / BlockSize;
  const int ArraySizeInBytes = ArraySize * sizeof(int);

  int *hostArray;
  cudaMallocHost(&hostArray, ArraySizeInBytes);

  // Allocate memory for the results
  int *squares, *cubes;
  cudaMallocHost(&squares, ArraySizeInBytes);
  cudaMallocHost(&cubes, ArraySizeInBytes);

  // Initialize array elements to 10
  for (int i = 0; i < ArraySize; i++) {
    hostArray[i] = 10;
  }
  // Create a stream for each operation
  cudaStream_t squareStream, cubeStream;
  cudaStreamCreateWithFlags(&squareStream, cudaStreamNonBlocking);
  cudaStreamCreateWithFlags(&cubeStream, cudaStreamNonBlocking);
  int *inputArrSquares;
  cudaMallocAsync(&inputArrSquares, ArraySizeInBytes, squareStream);
  int *inputArrCubes;
  cudaMallocAsync(&inputArrCubes, ArraySizeInBytes, cubeStream);
  int *devSquaredArray;
  cudaMallocAsync(&devSquaredArray, ArraySizeInBytes, squareStream);
  int *devCubedArray;
  cudaMallocAsync(&devCubedArray, ArraySizeInBytes, cubeStream);
  // warm-up kernels
  computeSquares<<<GridSize, BlockSize, 0, squareStream>>>(inputArrSquares,
                                                           devSquaredArray, true);
  computeCubes<<<GridSize, BlockSize, 0, cubeStream>>>(inputArrCubes,
                                                       devCubedArray, true);

  cudaDeviceSynchronize();

  // issue work depth-first
  cudaMemcpyAsync(inputArrSquares, hostArray, ArraySizeInBytes,
                  cudaMemcpyHostToDevice, squareStream);
  computeSquares<<<GridSize, BlockSize, 0, squareStream>>>(inputArrSquares,
                                                           devSquaredArray);
  cudaMemcpyAsync(squares, devSquaredArray, ArraySizeInBytes,
                  cudaMemcpyDeviceToHost, squareStream);
  cudaMemcpyAsync(inputArrCubes, hostArray, ArraySizeInBytes,
                  cudaMemcpyHostToDevice, cubeStream);
  computeCubes<<<GridSize, BlockSize, 0, cubeStream>>>(inputArrCubes,
                                                       devCubedArray);
  cudaMemcpyAsync(cubes, devCubedArray, ArraySizeInBytes,
                  cudaMemcpyDeviceToHost, cubeStream);

  // Synchronize ALL work on the device
  cudaDeviceSynchronize();

  // Sanity check
  for (int i = 0; i < ArraySize; i++) {
    assert(squares[i] == 100);
    assert(cubes[i] == 1000);
  }
}
# nvcc -o t395 t395.cu -arch=sm_89
# nsys nvprof --print-gpu-trace ./t395
WARNING: t395 and any of its children processes will be profiled.

Generating '/tmp/nsys-report-0fa8.qdstrm'
[1/3] [========================100%] report74.nsys-rep
[2/3] [========================100%] report74.sqlite
[3/3] Executing 'cuda_gpu_trace' stats report

 Start (ns)   Duration (ns)  CorrId  GrdX  GrdY  GrdZ  BlkX  BlkY  BlkZ  Reg/Trd  StcSMem (MB)  DymSMem (MB)  Bytes (MB)  Throughput (MBps)  SrcMemKd  DstMemKd     Device      Ctx  Strm                 Name
 -----------  -------------  ------  ----  ----  ----  ----  ----  ----  -------  ------------  ------------  ----------  -----------------  --------  --------  -------------  ---  ----  ----------------------------------
 649,507,432          1,376     127     8     1     1   128     1     1       16         0.000         0.000                                                     NVIDIA L4 (0)    1    13  computeSquares(int *, int *, bool)
 649,553,001          1,248     128     8     1     1   128     1     1       16         0.000         0.000                                                     NVIDIA L4 (0)    1    14  computeCubes(int *, int *, bool)
 649,577,576            609     130                                                                                0.004          6,725.779  Pinned    Device    NVIDIA L4 (0)    1    13  [CUDA memcpy HtoD]
 649,587,689        223,072     131     8     1     1   128     1     1       16         0.000         0.000                                                     NVIDIA L4 (0)    1    13  computeSquares(int *, int *, bool)
 649,606,569            511     133                                                                                0.004          8,015.655  Pinned    Device    NVIDIA L4 (0)    1    14  [CUDA memcpy HtoD]
 649,613,897        242,272     134     8     1     1   128     1     1       16         0.000         0.000                                                     NVIDIA L4 (0)    1    14  computeCubes(int *, int *, bool)
 649,815,593          1,216     132                                                                                0.004          3,368.419  Device    Pinned    NVIDIA L4 (0)    1    13  [CUDA memcpy DtoH]
 649,857,577            960     135                                                                                0.004          4,266.664  Device    Pinned    NVIDIA L4 (0)    1    14  [CUDA memcpy DtoH]

Generated:
    /.../report74.nsys-rep
    /.../report74.sqlite
#

(CUDA 12.2, L4 GPU)

I caught the “lazy loading” in the CUDA overhead tab in my Nsight systems GUI and now it makes sense why there was no overlap. Thank you for the response!