In my CUDA code, sometimes within a block, some warps finish earlier while others are still computing. I’m wondering if the warps that finish first might waste computational resources. Should I create a dynamic resource pool to keep idle warps busy? However, I’m also concerned because there are only four warp schedulers in an SM, and not all warps are active simultaneously. So when a warp finishes, it doesn’t consume resources anymore. Is it necessary to implement job stealing/work balancing within a block?
While the concept of occupancy exists in CUDA, in reality, the four warp schedulers on an SM don’t actually allow for 16 warps to run simultaneously if there are 512 threads. They just take turns executing in a way that they mask each other’s latency, like alternating between reading and computing. So if some warps finish early, it mainly affects the degree of this latency masking. In cases where computation is dense, it probably doesn’t matter much, right?
Sorry, the title should be “within a block” but not “within a warp”…