This might have run into a stack overflow exception.
To debug issues like these you can do the following:
Enable all exceptions:
Enable printing on all launch indices:
(When you know which launch index is failing, e.g. (x,y), you can limit the printout to that launch index for individual printf debugging:
Only enable that application code above when debugging! It affects the emitted PTX code. Do not leave this enabled when benchmarking!
Provide an exception program per entry-point which decodes and prints the exception code. I use this:
rtDeclareVariable(uint2, launchIndex, rtLaunchIndex, );
RT_PROGRAM void exception()
const unsigned int code = rtGetExceptionCode();
rtPrintf("Exception code 0x%X at (%d, %d)\n", code, launchIndex.x, launchIndex.y);
You could also output a special color to your output buffer in this case to highlight the error visually.
Then run your application from a command prompt and see if there are exceptions printed by this exception program.
You’ll find the enum values for the predefined exceptions in the optix_declarations.h header, for example RT_EXCEPTION_STACK_OVERFLOW = 0x3FC.
If that’s it, increase the stack size m_context->setStackSize(size); until this doesn’t happen anymore,
or reduce your per ray payload size,
or change your algorithm from being recursive to iterative.