Hello,
Resource usage of a CUDA kernel, as seen by using --ptxas-options=-v, is
for Compute Capability 2.0. But for such devices, instead of using an odd number of registers, one might as well use the next even number of registers. I believe the “4+0 bytes lmem” is for register spill over – if indeed the 4 bytes of lmem are for register spillover, why didn’t nvcc use 26 registers – thereby eliminating the needed for any bytes in lmem?
.
My first guess would be it is for a variable whose address is taken somewhere.
Are you using trigonometric functions? I think their implementations also use local memory. Anyway it should not be of too much concern on 2.0 devices as it will be cached.
.
The kernel is very simple (essentially, just some adds and multiplies) – and it is a big issue because performance drops drastically:
+------------------------------------+
| param | threads/block | blocks/grd |
|-------+---------------+------------|
| 2 | 16 x 16 x 1 | 32768 |
| 4 | 16 x 16 x 1 | 16384 |
| 8 | 16 x 16 x 1 | 8192 |
| 16 | 16 x 16 x 1 | 4096 |
| | | |
+------------------------------------+
+------------------------------------------+
| param | regs | mem | ms | GByt/s |
|-------+------+----------+-------+--------|
| 2 | 25 | 76 cm[0] | 263.7 | 130.8 |
| 4 | 31 | 76 cm[0] | 221.8 | 155.5 |
| 8 | 41 | 76 cm[0] | 184.6 | 186.7 |
| 16 | 25 | 4+0 lm, | 655.7 | 52.6 |
| | | 76 cm[0] | | |
+------------------------------------------+
I can’t believe the lmem usage to be the reason for that drastic drop in throughput, as the cache should help degrading gracefully.
Anyway, if the lmem usage really comes from the compiler trying to save registers, you might be able to change that using the launch_bounds() directive. See appendix B.17 of the Programming Guide for details.
post your kernell, maybe you are using arrays etc.