I found multiple instances of answer for my query. But, still its very confusing for me. Sorry to ask again here.
I am running “deviceQuery” in google Colab. And below is the response from GPU.
./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "Tesla T4" CUDA Driver Version / Runtime Version 11.2 / 11.2 CUDA Capability Major/Minor version number: 7.5 Total amount of global memory: 15110 MBytes (15843721216 bytes) (40) Multiprocessors, ( 64) CUDA Cores/MP: 2560 CUDA Cores GPU Max Clock rate: 1590 MHz (1.59 GHz) Memory Clock rate: 5001 Mhz Memory Bus Width: 256-bit L2 Cache Size: 4194304 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total shared memory per multiprocessor: 65536 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 3 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Enabled Device supports Unified Addressing (UVA): Yes Device supports Managed Memory: Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 4 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.2, CUDA Runtime Version = 11.2, NumDevs = 1 Result = PASS
I know that,
Maximum number of threads in a thread-block is 1024 (Even in 3D this is max. For example, (1024,1,1) is valid; (32,32,1) is valid, but (33,33,1) is not valid and so on)
I also know that, there is a large pool of blocks possible in grid. Which is (2147483647, 65535, 65535)
Here is my question.
What is the maximum number of threads that can simultaneously run in GPU?
** Is it 2560 threads at given an instant of time? Because we have 2560 cores [(40) Multiprocessors, ( 64) CUDA Cores/MP]
If my above explanation is correct, then, if I launch a kernel with <<<10000, 1024>>>, will the threads will be time-shared? Meaning, We total have 10240000 to execute; and GPU will execute 4000 times (10240000 /2560)?