# cuda profiler result of same program under GTX295 and GTX470 register per thread, local load, local **URL:** \ **Category:** CUDA Programming and Performance\ **Created:** [May 3, 2010, 4:56pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421 "2010-05-03T16:56:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![springc](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@springc](https://forums.developer.nvidia.com/u/springc)\ **Post date:** [May 3, 2010, 4:56pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/1 "2010-05-03T16:56:41Z")
Since my application runs slower in gtx 470, I thought about run it in profiler. The binary is compiled months ago with cuda 2.1 (or 2.2). However, the items showed in the profiler is changed and the values changed as well. GTX 470: method gputime cputime occupancy gridSizeX gridSizeY blockSizeX blockSizeY blockSizeZ dynSmemPerBlock staSmemPerBlock registerPerThread streamID memTransferSize memtransferhostmemtype local\_load local\_store gld\_request gst\_request shared\_load shared\_store branch warps\_launched active\_cycles sm\_cta\_launched l1\_global\_load\_hit divergent\_branch l1\_global\_load\_miss inst\_issued inst\_executed threads\_launched active\_warps Kernel 3821.92 3834.11 0.667 127 1 128 1 1 0 0 21 0 0 55476 55476 55476 55476 55476 55476 19512 55476 55476 9 16584 0 92442 1786040 1707588 1152 46693123 GTX 295: method gputime cputime occupancy gridSizeX gridSizeY blockSizeX blockSizeY blockSizeZ dynSmemPerBlock staSmemPerBlock registerPerThread streamID memTransferSize memtransferhostmemtype branch divergent\_branch instructions warp\_serialize cta\_launched local\_load local\_store gld\_32b gld\_64b gld\_128b gst\_32b gst\_64b gst\_128b tex\_cache\_hit tex\_cache\_miss Kernel 2798.69 2813.74 1 127 1 128 1 1 0 44 15 0 0 33168 0 784324 0 13 624 1248 72192 86016 82944 24064 1207584 27648 0 0 So the GTX 295 are about 30% faster. It only use 15 registers per thread but it also use 44 static shared memory per block, whereas the GTx 470 use 21 registers and 0 static share memory per block. GTX 470 profiler report doesn’t have those gld and gst, which I believe is due to the L1 cache (But how about loading data from device memory into L2 cache? is there anywhere to show how many data get loaded into cache?) For our code, we have good coalesced access (but not necessary aligned), as you can see from the gld and gst field in GTX 295. However, those data are used only once and the access pattern are scattered so L1 cache doesn’t help us, as shown in the GTX 470 result where we have 92442 L1 misses but only 16584 L1 hits (probably at sacrfice of loading extra data and underuse the bandwidth). What concerns me is the 55476 local load and local store in GTx 470 vs. only 624 local load and 1248 local store in GTX 295. I suspect it is what makes our program slower. So I am wondering, what makes the local memory load and store increase? note that the registers per thread also increased, what causes that? Remember we didn’t recompile the code, it is still the one we compiled weeks ago using cuda2.1. Why do we get different register usage per thread running on different card? ---
**Author:** ![Lev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Lev](https://forums.developer.nvidia.com/u/Lev)\ **Post date:** [May 3, 2010, 5:14pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/2 "2010-05-03T17:14:43Z")
Do you consider use more recent version of sdk? Could you rewrite report in line way? Btw, do you use special function units? And what is local store? ---
**Author:** ![springc](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@springc](https://forums.developer.nvidia.com/u/springc)\ **Post date:** [May 3, 2010, 9:41pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/3 "2010-05-03T21:41:05Z")
Sorry for the messy format of the profiler output. GTX 470 method gputime cputime occupancy gridSizeX gridSizeY blockSizeX blockSizeY blockSizeZ Kernel\_ 3821.92\_3834.11\_0.667\_\_\_\_\_127 \_\_\_\_\_\_1\_\_\_\_\_\_\_128\_\_\_\_\_\_\_\_1\_\_\_\_\_\_\_1 dynSmemPerBlock staSmemPerBlock registerPerThread streamID memTransferSize memtransferhostmemtype 0\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_0\_\_\_\_\_\_\_\_\_\_\_\_\_21\_\_\_\_\_\_\_\_\_\_\_\_0\_\_\_\_\_\_\_\_\_\_ 0 local\_load local\_store gld\_request gst\_request shared\_load shared\_store branch warps\_launched active\_cycles sm\_cta\_launched 55476\_\_\_\_\_\_55476\_\_\_\_\_55476\_\_\_\_\_55476\_\_\_\_\_55476\_\_\_\_\_\_55476\_\_\_ \_19512\_\_\_\_\_\_55476\_\_\_\_\_\_\_\_55476\_\_\_\_\_\_9 l1\_global\_load\_hit divergent\_branch l1\_global\_load\_miss inst\_issued inst\_executed threads\_launched active\_warps 16584\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_0\_\_\_\_\_\_\_\_\_\_\_\_\_\_92442\_\_\_\_\_\_\_\_1786040\_\_\_17 07588\_\_\_\_\_\_\_\_\_1152\_\_\_\_\_\_46693123 GTX 295: method gputime cputime occupancy gridSizeX gridSizeY blockSizeX blockSizeY blockSizeZ Kernel\_\_2798.69\_2813.74 \_\_\_1\_\_\_\_\_\_127\_\_\_\_\_\_\_1\_\_\_\_\_\_128\_\_\_\_\_\_\_1\_\_\_\_\_\_\_\_\_1 dynSmemPerBlock staSmemPerBlock registerPerThread streamID memTransferSize memtransferhostmemtype _ **0\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_44\_\_\_\_\_\_\_\_\_\_\_\_\_15\_\_\_\_\_\_\_\_\_\_\_\_0** _\_\_\_ \_\_0 branch divergent\_branch instructions warp\_serialize cta\_launched local\_load local\_store 33168\_\_\_\_\_\_\_0\_\_\_\_\_\_\_\_\_\_784324\_\_\_\_\_\_0\_\_\_\_\_\_\_\_\_\_\_\_13\_\_\_\_\_\_\_624 \_\_\_\_\_1248 gld\_32b gld\_64b gld\_128b gst\_32b gst\_64b gst\_128b tex\_cache\_hit tex\_cache\_miss 72192\_\_86016\_\_82944\_\_\_24064\_1207584\_27648\_\_\_\_\_\_\_0\_\_\_\_\_\_\_\_\_\_\_ 0 Hope this is better. We do use a few exponential calculation. I have no idea what that Local\_load and Local\_store in cuda profiler means. Does it mean it saves local variable instead of keep it in register? We will compile with toolkit 3.0, need to get visual studio updated first. > [@](#): > > Do you consider use more recent version of sdk? Could you rewrite report in line way? Btw, do you use special function units? And what is local store? ---
**Author:** ![Lev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Lev](https://forums.developer.nvidia.com/u/Lev)\ **Post date:** [May 3, 2010, 10:09pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/4 "2010-05-03T22:09:50Z")
It will be interesting co compare. But you can compile with fast math switch to get rid of exp functions to check if they are bottle neck and using local memory. Looks lime you have same number of threads in gtx295 and gtx 470. Do you use a lot of divisions and square roots? ---
**Author:** ![springc](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@springc](https://forums.developer.nvidia.com/u/springc)\ **Post date:** [May 5, 2010, 6:26pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/5 "2010-05-05T18:26:15Z")
No, that doesn’t quite work. I only did a few exp() and divisions. After use the fast match switch, I can see the instruction decreased and kernel runs a little bit faster, but only 1 or 2 percent at most. > [@](#): > > It will be interesting co compare. But you can compile with fast math switch to get rid of exp functions to check if they are bottle neck and using local memory. > > Looks lime you have same number of threads in gtx295 and gtx 470. Do you use a lot of divisions and square roots? ---
**Author:** ![springc](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@springc](https://forums.developer.nvidia.com/u/springc)\ **Post date:** [May 6, 2010, 9:32pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/6 "2010-05-06T21:32:06Z")
After remove portion of code inside kernel to diagnose, we finally discovered that it is because some calculation is performed in double precision. We declared every variable to be float, but there are some constant that we hard coded, like 1.0. Which seems to compiled as double precision calculation. Interestingly, it doesn’t impact the performance in GTX 295. After we change all numerical notation with 1.0f. It calculates much faster. Now the 470 can is about 15% faster than a single GTX 295 GPU. Now, I am wondering, is there any command switch to force all floating point calculation to be single precision? I know the arch=sm11 will do this, but I worry it will also turn off other capabilities that is desirable. ---
**Author:** ![Lev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Lev](https://forums.developer.nvidia.com/u/Lev)\ **Post date:** [May 6, 2010, 9:39pm UTC](https://forums.developer.nvidia.com/t/cuda-profiler-result-of-same-program-under-gtx295-and-gtx470-register-per-thread-local-load-local/16421/7 "2010-05-06T21:39:19Z")
Btw, this float constant thing was mentioned in programming guide. Ah, you may try to turn on flush to zero mode, as it was default in gt200, maybe it helps too.