Since single-precision operations are at least as fast as double precision’s, I can take the performance advantage by writing code that do not require double precision operations in single precision.
Question: How can I know if it is safe to write a CUDA application in single precision? Or, how can I know if using single precision operation in my application is not going to generate computational errors that are avoidable using double precision operations? Has any research been done regarding this topic?
For all types of (PDE) problems where the condition number depends on the mesh width, single is bound to fail, but a mixed precision iterative refinement procedure (few double precision updates, bulk of the work in fast single) gives correct results in less time if the problem is “not too ill-conditioned” in single precision.
In my experiences, single precision is almost always good enough. The question then becomes, when is single precision not safe, or “where do I really need double precision”.
In very general terms (and something that doesn’t apply to Dominik’s work, showing that there are sometimes exceptions), the worst cases where single precision can hurt are:
subtraction of two numbers very near each other (cancellation error)
addition of a large number plus a small one (round off error)
As you will find the the link I gave, these cases hurt in double precision too: double precision is not a cure-all. They just hurt a little bit less because you’ve got more sig. figs.