Optix measures the time a warp has started against the time specified in timeout. Problems arise when for example you have a launch index that can take 0.5 seconds, the timeout is 1.8 seconds and the current elapsed time is 1.6 seconds. OptiX says it’s OK to proceed (1.6 < 1.8), but you get a timeout when you wanted to use 2.1 seconds.
Generally you want to set the timeout to be TDR - MaxTimeForSingleLaunchIndex. In the above example, you would set the timeout at 1.5 seconds if your TDR is 2 seconds.
Note that you can also adjust the TDR in your windows registry. There are instructions of how to do so in the CUDA Toolkit documentation.
In addition a Tesla device or a Quadro device put in TCC mode will not have the TDR limitation. I use a Tesla device, and I love it for OptiX work.