First kernels launch can't run in parallel?

I’m very new in cuda programming. In a basicly kernels running, I launch 2 kernels like this: kernel1<<<4,256,0,stream1>>>();kernel2<<<4,256,0,stream2>>>();

Their code is exactly the same especialy their name and stream.They perform simple addition operations through a for loop and write nothing to memory. And no cuda event or sync function I used in context. I only use cudaDeviceSynchronize();before cudafree.

For stream1 and 2, they were created like this:

cudaStreamCreateWithFlags(&stream1,cudaStreamNonBlocking);

Then, there’s no overlapping in Nsight System’s report!

My os is ubuntu24.04. My GPU is GTX1060 3G, and 4 blocks with 256 threads per block can’t fill the cuda core at all.

And then I tried same kernel in different streams, overlapping has shown.

I just change the “kernel2” to “kernel1” in launch line. It seems some basic concept I ignored, could anyone tell me what should I know. Thank all.

If I understood you correctly, it is as intended. Each stream runs one operation at a time. It is like a PC with different threads. To run something in parallel, you need multithreading. For Cuda to run several kernels in parallel you need multiple streams. Within one stream the execution is sequential (however the Cuda calls may return early to the CPU). It is like work queues.

(This was on the level of kernels. Each kernel itself has parallel execution of the Cuda threads.)

You also need to avoid lazy loading effects.

So the very first time you launch these kernels:

kernel1<<<4,256,0,stream1>>>();
kernel2<<<4,256,0,stream2>>>();

it’s expected that there will be no overlap.

Then, in the same program, if you launch them again:

kernel1<<<4,256,0,stream1>>>();
kernel2<<<4,256,0,stream2>>>();

you may witness overlap at that point.

Yes! I see overlap when kernels were launched again. Thank you! So it’s correctly that any stream’s first kernel’s first launch along with lazy loading effects? I haven’t find this in NV’s document yet.

Thank you for reply. May be there’s a lazy loading effect in every launch for stream’s first kernel’s first launch.

It’s not “any stream”. Its the first time a kernel is launched in your app. The stream it is launched into does not matter regarding lazy loading.

Thank you for reply. I add a “firstkernel” and it was launched in both stream1&2 before kernel1&2, if I understood you correctly, the firstkernel is “the first time a kernel is launched” in my app. The firstkernel do the same thing as kernel1&2 with smaller workload. And no overlap between kernel1&2. Do I missunderstand your reply?

You misunderstood Robert_Crovella’s answer. Any kernel which is launched for the first time is affected by lazy loading. It is the first time launching kernel1 and kernel2, so no overlap.

Lazy loading is described here: CUDA C++ Programming Guide — CUDA C++ Programming Guide

You can disable it by setting the environment variable `CUDA_MODULE_LOADING=EAGER`

Thank you a lot, this truly help me.