I’m very new in cuda programming. In a basicly kernels running, I launch 2 kernels like this: kernel1<<<4,256,0,stream1>>>();kernel2<<<4,256,0,stream2>>>();
Their code is exactly the same especialy their name and stream.They perform simple addition operations through a for loop and write nothing to memory. And no cuda event or sync function I used in context. I only use cudaDeviceSynchronize();before cudafree.
If I understood you correctly, it is as intended. Each stream runs one operation at a time. It is like a PC with different threads. To run something in parallel, you need multithreading. For Cuda to run several kernels in parallel you need multiple streams. Within one stream the execution is sequential (however the Cuda calls may return early to the CPU). It is like work queues.
(This was on the level of kernels. Each kernel itself has parallel execution of the Cuda threads.)
Yes! I see overlap when kernels were launched again. Thank you! So it’s correctly that any stream’s first kernel’s first launch along with lazy loading effects? I haven’t find this in NV’s document yet.
Thank you for reply. I add a “firstkernel” and it was launched in both stream1&2 before kernel1&2, if I understood you correctly, the firstkernel is “the first time a kernel is launched” in my app. The firstkernel do the same thing as kernel1&2 with smaller workload. And no overlap between kernel1&2. Do I missunderstand your reply?
You misunderstood Robert_Crovella’s answer. Any kernel which is launched for the first time is affected by lazy loading. It is the first time launching kernel1 and kernel2, so no overlap.