In my quest for optimal GPU performance, I’ve noticed a strange occurance.
On my M2090, the first three times I execute my program I get great performance and using [font=“Courier New”]nvidia-smi[/font] I see the GPU Utilization at 90% and the code only takes 10 seconds to run. However, on the 4th run and ever after, I only see 55% utilization and the code takes 15 seconds to run. I’ve used cuda-memcheck but it doesn’t show anything amiss.
If I unload the CUDA kernel module and reload it. I get back to the 90% utilization for three executions and then it’s back down to 55% ever after.
Has anyone else experienced this before or know what’s going on? This doesn’t happen on my C1060 nor my GTX 460, all using the same driver versions (295.49). My M2090 devices are in a SuperMicro SuperServer 1026GT-TF-FM205