I’m experimenting with denoised path tracing using the basic code from the “optixDenoiser” sample scene. Optix works amazing well but I’m hitting some instability when there’s lots of objects to trace.
try CUDA 9.0 (or even 8.0) and device driver: 388.59 and TDR enabled (with long delay times)
I had many crashs with CUDA 9.1 in combination with OptiX 5.0.0 on that path tracer + denoiser sample (and a deactivated TDR) but since using CUDA 9.0 and driver 388.59 and enabled TDR no more crashs yet.
In release notes [url]https://developer.nvidia.com/designworks/optix/downloads/5.0.0/releasenotes[/url]
Take a look at "Development Environment Requirements " and at “known issues”.
NOTE: when using CUDA 8.0 the MDL part may cause trouble again, see your older question [url]Can't compile the MDL samples using VS 2015 - OptiX - NVIDIA Developer Forums