Please understand my low command of english :(
There are some information while developing CUDA application.
When allocate memory and calculate on a GPU, if using memory is over 90%(or 70~60% on mid level GPU) of maximum memory, GPU calculation is being unstable.
It is because of GPU using base memory and it’s size is changing continuously.
When I allocate memory with enough free memory, there were no problem.
I want to share some know how and information about GPU computing development.
Check every cudaMalloc()'s return value for an error. Just like CPU, if you’re allocating large amounts of memory, new allocations may fail because the remaining memory is fragmented enough that you there are no large single ranges of memory available. The return code of the cudaMallocs will tell you if this is the case. If it is, there are several ways to reduce it, including rearranging mallocs from large to small, making one HUGE malloc and suballocating it yourself, or using host memory mapping to store your largest data if it’s not accessed often.
If all of your memory mallocs are succeeding but the compute is still unstable… are you running an application with any OpenGL/DirectX rendering? If so, the display portion of the GPU always gets memory priority and is allowed to forcibly terminate CUDA compute to steal its memory. This problem isn’t so common anymore with multi-gigabyte cards, but if you are running graphics, it might still explain your symptoms.