First, read the OptiX Release Notes before setting up a development environment.
You need to be a little careful about which versions of OptiX, CUDA Toolkit, and host compiler work together.
The newest things which work together would be:
- Under Windows: Microsoft Visual Studio 2013
If you don’t have a Professional version, the MSVS 2013 Community edition should work, not any 2015 or Express edition! CUDA 7.5 doesn’t support these. (There was a thread on this forum where I explained this before, with a link to the compiler which worked for another user. https://devtalk.nvidia.com/default/topic/891038/?comment=4718168 )
- OptiX 3.9.0
- CUDA Toolkit 7.5
Omit the display driver installation in that, that’s only required when it’s shipping before there is an official display driver supporting a specific CUDA version. Just use a recent enough display driver.
To learn OptiX programming I normally recommend starting like this:
For adding own samples inside the OptiX SDK path, please read the topmost CMakeLists.txt in the OptiX 3.9.0/SDK folder of your installation.
If you have programmed GLSL fragment shaders before, that’s basically what goes into the closest hit programs in OptiX. Surface shading is mainly about implementing sampling and evaluation of bi-directional scattering functions. There are a lot of books and papers about that.
If you plan to shop for a graphics board for ray tracing, I would recommend a beefy Kepler GPU board (GK110 or GK104), or better a new Maxwell second generation GPU (GM2xx). Wikipedia has a list of boards and GPU versions. The more VRAM, the better to handle acceleration structures of bigger scenes.