Maximum number of threads in a GPU

Accelerated Computing CUDA CUDA Programming and Performance
1

I found multiple instances of answer for my query. But, still its very confusing for me. Sorry to ask again here.

I am running “deviceQuery” in google Colab. And below is the response from GPU.

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Tesla T4"
  CUDA Driver Version / Runtime Version          11.2 / 11.2
  CUDA Capability Major/Minor version number:    7.5
  Total amount of global memory:                 15110 MBytes (15843721216 bytes)
  (40) Multiprocessors, ( 64) CUDA Cores/MP:     2560 CUDA Cores
  GPU Max Clock rate:                            1590 MHz (1.59 GHz)
  Memory Clock rate:                             5001 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        65536 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1024
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 4
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.2, CUDA Runtime Version = 11.2, NumDevs = 1
Result = PASS

I know that,

  • Maximum number of threads in a thread-block is 1024 (Even in 3D this is max. For example, (1024,1,1) is valid; (32,32,1) is valid, but (33,33,1) is not valid and so on)

  • I also know that, there is a large pool of blocks possible in grid. Which is (2147483647, 65535, 65535)

Here is my question.

  • What is the maximum number of threads that can simultaneously run in GPU?
    ** Is it 2560 threads at given an instant of time? Because we have 2560 cores [(40) Multiprocessors, ( 64) CUDA Cores/MP]

  • If my above explanation is correct, then, if I launch a kernel with <<<10000, 1024>>>, will the threads will be time-shared? Meaning, We total have 10240000 to execute; and GPU will execute 4000 times (10240000 /2560)?

Thanks!!
Aravind

1 Like
2

The best number for this is the maximum complement of threads per SM (related to occupancy) times the number of SMs in your GPU. Without considering occupancy limiters in a specific code (e.g. registers per thread, shared memory usage, etc.) the maximum number of threads per SM is a hardware limit that is in your deviceQuery output as “Maximum number of threads per multiprocessor” (it is also documented in the programming guide.) If your code has occupancy characteristic (register usage, shared memory usage, etc.) that restrict the occupancy to a smaller number, then use that.

When you launch more than the maximum instantaneous capacity, the additional threads are waiting in queue. They are not time-shared. They wait until some of the first group of threads retire, then they begin executing with the freed-up resources made available when other threads retire.

3

Thanks Robert. I have a small confusion. When you say “maximum complement of threads”, it is equivalent to threads only right? Meaning, in the above example,

Maximum number of threads per multiprocessor:  1024

and Number of Multiprocessors = 40

Hence, number of instantaneous threads which can run in a GPU (ideally) = 1024 * 40 = 40960

Correct?

I understand your suggestion regarding occupancy limiters.

4

Yes, correct.

1 Like
5

Thanks @Robert_Crovella

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.