# Disabling cache on Fermi architectures Try to disable L1 and L2 **URL:** \ **Category:** CUDA Programming and Performance\ **Created:** [September 10, 2010, 12:48pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722 "2010-09-10T12:48:28Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![CNugteren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@CNugteren](https://forums.developer.nvidia.com/u/CNugteren)\ **Post date:** [September 10, 2010, 12:48pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/1 "2010-09-10T12:48:28Z")
In relation to some research, I am trying to disable caches on my Fermi card (GTX470). So far, I succeeded in disabling the complete level 1 caches by using the following compiler flag for nvcc: -Xptxas -dlcm=cg This decreases the performance, so I assume the level 1 cache is actually disabled. However, there is no manual or help on the ‘dlcm’ option, other than that it supports the values ca (enable L1) and cg (disable L1). Secondly, I tried to disable the L2 cache, but so far without success. Is there any information on this topic available? A solution for me would be welcome in one of the following ways: - As a compiler flag (similar to disabling L1 cache) - As a function in the CUDA (host/kernel) code - As a workaround (tricking the compiler not to cache) Input is welcome! ---
**Author:** ![MisterAnderson42](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@MisterAnderson42](https://forums.developer.nvidia.com/u/MisterAnderson42)\ **Post date:** [September 10, 2010, 1:40pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/2 "2010-09-10T13:40:37Z")
As far as I know, you cannot disable L2. On various Fermi architecture presentations, it has been stated that every memory transaction goes through L2. (i.e., this one: [url=“[https://hub.vscse.org/resources/287/download/cuda\_fermi\_overview\_DK.pdf](https://hub.vscse.org/resources/287/download/cuda_fermi_overview_DK.pdf)”][https://hub.vscse.org/resources/287/downloa...overview\_DK.pdf[/url]](https://hub.vscse.org/resources/287/downloa...overview_DK.pdf%5B/url%5D) slide 12 - there is a video too [url=“[http://groups.google.com/group/vscse-many-core-processors-2010/web/course-presentations](http://groups.google.com/group/vscse-many-core-processors-2010/web/course-presentations)”][http://groups.google.com/group/vscse-many-...e-presentations[/url]](http://groups.google.com/group/vscse-many-...e-presentations%5B/url%5D) ) You might check the PTX manual and see if there is even an instruction to read from DRAM and not from L2. ---
**Author:** ![MisterAnderson42](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@MisterAnderson42](https://forums.developer.nvidia.com/u/MisterAnderson42)\ **Post date:** [September 10, 2010, 1:40pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/3 "2010-09-10T13:40:37Z")
As far as I know, you cannot disable L2. On various Fermi architecture presentations, it has been stated that every memory transaction goes through L2. (i.e., this one: [url=“[https://hub.vscse.org/resources/287/download/cuda\_fermi\_overview\_DK.pdf](https://hub.vscse.org/resources/287/download/cuda_fermi_overview_DK.pdf)”][https://hub.vscse.org/resources/287/downloa...overview\_DK.pdf[/url]](https://hub.vscse.org/resources/287/downloa...overview_DK.pdf%5B/url%5D) slide 12 - there is a video too [url=“[http://groups.google.com/group/vscse-many-core-processors-2010/web/course-presentations](http://groups.google.com/group/vscse-many-core-processors-2010/web/course-presentations)”][http://groups.google.com/group/vscse-many-...e-presentations[/url]](http://groups.google.com/group/vscse-many-...e-presentations%5B/url%5D) ) You might check the PTX manual and see if there is even an instruction to read from DRAM and not from L2. ---
**Author:** ![CNugteren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@CNugteren](https://forums.developer.nvidia.com/u/CNugteren)\ **Post date:** [September 10, 2010, 2:07pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/4 "2010-09-10T14:07:22Z")
Thanks, that pointed me in the correct direction. From the PTX manual we have: > [@](#): > > .ca Cache at all levels, likely to be accessed again. > > .cg Cache at global level (cache in L2 and below, not L1). > > .cs Cache streaming, likely to be accessed once. > > .cv Cache as volatile (consider cached system memory lines stale, fetch again). (there is a more detailed description in the manual on page 109 (ptx\_isa\_20)). Luckily, these options translate to the -dlcm option for ptxas. When I try the ‘cs’ option, performance decreases again! From the description, I guess that this means we omit both caches: > [@](#): > > The ld.cs load cached streaming operation allocates global lines with evict-first policy in L1 > > and L2 to limit cache pollution by temporary streaming data that may be accessed once or > > twice. When ld.cs is applied to a Local window address, it performs the ld.lu operation. ---
**Author:** ![CNugteren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@CNugteren](https://forums.developer.nvidia.com/u/CNugteren)\ **Post date:** [September 10, 2010, 2:07pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/5 "2010-09-10T14:07:22Z")
Thanks, that pointed me in the correct direction. From the PTX manual we have: > [@](#): > > .ca Cache at all levels, likely to be accessed again. > > .cg Cache at global level (cache in L2 and below, not L1). > > .cs Cache streaming, likely to be accessed once. > > .cv Cache as volatile (consider cached system memory lines stale, fetch again). (there is a more detailed description in the manual on page 109 (ptx\_isa\_20)). Luckily, these options translate to the -dlcm option for ptxas. When I try the ‘cs’ option, performance decreases again! From the description, I guess that this means we omit both caches: > [@](#): > > The ld.cs load cached streaming operation allocates global lines with evict-first policy in L1 > > and L2 to limit cache pollution by temporary streaming data that may be accessed once or > > twice. When ld.cs is applied to a Local window address, it performs the ld.lu operation. ---
**Author:** ![Magorath](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Magorath](https://forums.developer.nvidia.com/u/Magorath)\ **Post date:** [September 13, 2010, 9:15am UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/6 "2010-09-13T09:15:47Z")
I’m using your topic as I’m currently doing the same analysis. I do have a small kernel which reads number in 7 different arrays, does simple arithmetic with the numbers and fills 12 other arrays in global memory. When I disable L1 cache, I get 20% better performance. I’m really struggling to interpret this correctly. Could someone maybe give some piece of advice ? If I disable the L1 and L2 cache, I however lower the performance by ~10%. I’m using a GTX 465 and all arrays are declared as restricted to help the compiler do its job. Any help would be really appreciated. ---
**Author:** ![Magorath](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Magorath](https://forums.developer.nvidia.com/u/Magorath)\ **Post date:** [September 13, 2010, 9:15am UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/7 "2010-09-13T09:15:47Z")
I’m using your topic as I’m currently doing the same analysis. I do have a small kernel which reads number in 7 different arrays, does simple arithmetic with the numbers and fills 12 other arrays in global memory. When I disable L1 cache, I get 20% better performance. I’m really struggling to interpret this correctly. Could someone maybe give some piece of advice ? If I disable the L1 and L2 cache, I however lower the performance by ~10%. I’m using a GTX 465 and all arrays are declared as restricted to help the compiler do its job. Any help would be really appreciated. ---
**Author:** ![MichealHou](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@MichealHou](https://forums.developer.nvidia.com/u/MichealHou)\ **Post date:** [May 23, 2011, 6:43am UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/8 "2011-05-23T06:43:20Z")
> [@](#): > > I’m using your topic as I’m currently doing the same analysis. > > I do have a small kernel which reads number in 7 different arrays, does simple arithmetic with the numbers and fills 12 other arrays in global memory. > > When I disable L1 cache, I get 20% better performance. I’m really struggling to interpret this correctly. Could someone maybe give some piece of advice ? > > If I disable the L1 and L2 cache, I however lower the performance by ~10%. > > I’m using a GTX 465 and all arrays are declared as restricted to help the compiler do its job. > > Any help would be really appreciated. maybe the L1 cache miss rate is too high. lots of time are wasted on check. ---
**Author:** ![hyqneuron](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hyqneuron/32/9902_2.png) [@hyqneuron](https://forums.developer.nvidia.com/u/hyqneuron)\ **Post date:** [May 23, 2011, 11:57am UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/9 "2011-05-23T11:57:32Z")
I’m not quite sure about the latency of cache operations. But another thing is that when L1 is disabled L1 starts issuing 32-byte cache line accesses instead of the default 128-byte lines. That saves some global memory bandwidth if you do not read/write to continuous regions. ---
**Author:** ![wlangdon](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@wlangdon](https://forums.developer.nvidia.com/u/wlangdon)\ **Post date:** [August 29, 2013, 3:19pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/10 "2013-08-29T15:19:31Z")
the nvcc -Xptxas -dlcm appears to be applied by the CUDA compiler to the kernel being compiled. But caches are global. When I start another kernel (compiled without -Xptxas) will the caches all revert to normal (ie their defaults). Thanks Bill ---
**Author:** ![njuffa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@njuffa](https://forums.developer.nvidia.com/u/njuffa)\ **Post date:** [August 29, 2013, 5:46pm UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/11 "2013-08-29T17:46:29Z")
-Xptxas -dlcm does not cause machine state to be changed. It changes the code generation, so a different flavor of load instructions for accessing global memory is generated. Only the global load instructions in a given compilation unit are affected. One can change the load behavior for individual global memory accesses by generating the desired load instruction flavor via inline PTX. ---
**Author:** ![wlangdon](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@wlangdon](https://forums.developer.nvidia.com/u/wlangdon)\ **Post date:** [August 30, 2013, 10:31am UTC](https://forums.developer.nvidia.com/t/disabling-cache-on-fermi-architectures-try-to-disable-l1-and-l2/18722/12 "2013-08-30T10:31:44Z")
Dear njuffa, Thank you very much for rapid and helpful reply. Bill