Cuda_cache_maxsize

Hi all!

We are running an open-source software, kilosort 2.5, which you can be download from here
GitHub - MouseLand/Kilosort: Fast spike sorting with drift correction for up to a thousand channels. We run kilosort on Matlab 2020a and with CUDA toolkit version 10.1. We installed and used the software without any problem but after 2 weeks of use the computer started to crash and reboot. Following an answer to this issue Kilosort2 crashing and restarting computer · Issue #75 · MouseLand/Kilosort · GitHub we set an environment variable "CUDA_CACHE_MAXSIZE " on the machine and set the cache to 1073741824. After this kilosort doesn’t failed again but I am afraid that I damaged the GPU somehow while I ran kilosort 2.5 before I increased CUDA_CACHE_MAXSIZE. We have another computer in the lab in which happened the same issue, but we didn’t change the CUDA_CACHE_MAXSIZE and the computer end up crashing at any moment and with a broken GPU. What am I doing increasing CUDA CACHE? Do you think that changing CUDA_CACHE_MAXSIZE could prevent GPU from breaking? How much can I increase CUDA_CACHE_MAXSIZE?
Thank you!