Yes, the trace depth is limiting the number of recursions. The recommendation to use iterative instead of recursive algorithms reduces the recursion amount to only one or mostly two trace levels. (Normally two if the closest hit program shoots a visibility ray for direct lighting.)
Most path tracers are written as iterative algorithms and continue in only one direction on a transparent surface (reflection or transmission) by sampling its probability in a progressive rendering algorithm.
If you are not rendering progressively but need a final frame in one launch that would require to store all possible ray directions during the surface sampling. That can be done with a ray stack in the ray generation program and additional secondary ray information at the per ray data (PRD) payload.
Basically implement a unidirectional path tracer and whenever there can be reflection and transmission events, handle one of the new sampled directions like the usual path continuation and store the secondary ray, path length, and material information on the PRD as well and push them onto the ray stack in the ray generation program and handle it later.
That would require as many rays in the ray stack as your maximum recursion depth.
This algorithm is heavy on the required memory bandwidth!
Rough outline of that as pseudo algorithm:
raygen()
{
initialize ray stack to be empty;
generate primary ray;
push primary ray onto ray stack;
while (ray stack is not empty)
{
pop ray segment information from ray stack;
init PRD with current ray segment;
while (current path length <= max path length)
{
optixTrace(current ray segment in PRD);
if (ray split event)
{
push secondary split ray from PRD onto stack;
}
accumulate returned radiance;
if (terminate path event)
{
break;
}
adjust path throughput;
++current path length;
}
}
store radiance into output buffer;
}
Right, that’s not a supported combination of OptiX, CUDA, and Visual Studio versions.
OptiX 5.1.1 doesn’t support CUDA 10.1 and the recommended CUDA 9.0 version doesn’t support MSVS 2019.
Please see The OptiX 5.1.1 release notes (below its download button) for the requirements:
[i]"Development Environment Requirements (for compiling with OptiX)
CUDA Toolkit 7.0, or 7.5, 8.0, 9.0
OptiX 5.1.1 has been built with CUDA 9, but any specified toolkit should work when compiling PTX for OptiX. If an application links against both the OptiX library and the CUDA runtime on Linux, it is recommended to use CUDA 9.0."[/i]
and the CUDA_Installation_Guide_Windows.pdf of CUDA 9.0 says that only up to MSVS 2017 is supported by that.
You should use OptiX 6.5.0 with CUDA 10.1 and MSVS 2019 instead.
I have a port of the OptiX Introduction sample #7 to OptiX 7 using the CUDA Runtime API and the CUDA Driver API and the plan is to make those available, but we’re currently looking for a different location for the OptiX advanced samples repository because the nvpro-samples are restructured and target OpenGL and Vulkan.
Make sure you have the CUDA_HOST_COMPILER set to the proper location.
It changed between MSVS 2015 and 2017 and the FindCUDA.cmake in the repository is outdated.
See links in the other thread.