Is there a particular order to how warps are statically assigned to SM sub partitions?

Accelerated Computing CUDA CUDA Programming and Performance
, ,
1

Yes. When a threadblock is deposited on a SM by the CWD/block scheduler, the warps in that threadblock are statically assigned to SMSPs (SM sub-partitions). Each sub-partition has a single warp scheduler, so this is like saying the warps are statically assigned to each of the warp schedulers. If there is only one warp scheduler, all warps will be assigned to that. If there are two warp schedulers, about half of the warps will be assigned to one (assuming the SM is empty) and about half will be assigned to the other. If there are 4 warp schedulers in the SM, and assuming an initially empty SM, then the warps will be distributed approximately 1/4 to each warp scheduler. Certain functional unit resources in a SM are also partitioned between the SMSPs. So a SM with 64 “cuda cores” and 4 warp schedulers means that each SMSP/warp scheduler actually only has 16 “cuda cores” to use or assign instructions to.

Is this still the case on consumer cards? (Most curious about Ampere, Ada and Blackwell).
If so is there any sort of order to how the warps are assigned? For example does each sub-partition get every fourth warp (or some other fixed distribution scheme) or is it affected by the amount of work already assigned to the SM? Is there any sort of typical pattern assuming the SM is initially empty? The reason I’m asking is because if there is indeed a pattern I would like to exploit it with gl_WarpIDNV to be able to find out what sub-partition a warp belongs to (which VK_NV_shader_sm_builtins doesn’t seem to expose) in order to try my hand at some dynamic work redistribution across the SM. Thanks!

2

Greg’s answer here covers some of your points.

3

Since Volta all GPUs (including consumer) have SMs comprising exactly 4 SMSPs. Some resources (registers, standard arithmetic, tensor cores) are dedicated to SMSPs, some resources (e.g. shared memory) are per SM.

For the distribution on SMSPs, the line from Greg’s post (see rs277’s post) will be helpful.

On most GPUs the lower 2-bits of the warp ID indicate the SM sub-partition.

In my experience the warps are typically distributed in round-robin fashion unto SMSPs, but there is no guarantee.

Hopper introduced the notion of warpgroups = warps with four contiguous ids. I am quite sure (but no internals released by Nvidia; and it could also have been specific to Hopper) that those have to be situated on different SMSPs to effectively combine the resources (Tensor Cores and bandwidth) of the 4 SMSPs. It also fits with the warp numbering mentioned by Greg.

If you have some warps dedicated to doing tensor core instructions or some other more specific usage of computation units, it is of advantage to distribute those among the SMSPs to utilize the full performance of the GPU. If you use 32 warps and 8 (every 4th) would use tensor cores, they should be assigned to e.g. warps 0-7. Not to e.g. warp 0, 4, 8, 12, … 28, which would land all on the same SMSP.