Did you read all of the other posts linked in the related topics recursively?
When reaching the closest hit program, right, that ray ends there.
At that point your closest hit program would need to handle the state machine.
In an iterative path tracer you would return back to the ray generation program afterwards, which then shoots another ray starting from that surface hit point into a direction you calculated with a method to not hit the same primitive again, normally by adding a scene related epsilon to the ray t_min interval start.
Similarly for a recursive program which would shoot the next ray from within the closest hit program.
(Don’t do that if you can work iteratively!)
Depending on what you’re actually trying to achieve (exact algorithmic problem description?) you might need to filter out back face hits.
Something like this for the iterative case.
(Pseudo code! There would need to be some more end conditions for the path termination.)
There is no any_hit program needed at all for this (unless primitives have coutouts done with anyhit tests).
// Inside the closest hit program:
// Setup the next ray segment origin, direction, distance for the iterative path tracer.
//...
// Handle the state machine checking the expected order of hits.
switch (prd.state)
{
case STATE_START:
prd.state = (objectID == A) ? STATE_HIT_A : STATE_NO_MATCH;
break;
case STATE_HIT_A:
prd.state = (objectID == B) ? STATE_HIT_AB : STATE_NO_MATCH;
break;
case STATE_HIT_AB:
prd.state = (objectID == C) ? STATE_MATCH_ABC : STATE_NO_MATCH; // End state.
break;
//default:
// Do nothing, keep the last state for analysis in raygen program.
}
// Inside the miss program.
RT_PROGRAM miss()
{
prd.state = (prd.state == STATE_HIT_AB) ? STATE_MATCH_ABMiss : prd.state; // Keep the previous state in case that was STATE_MATCH_ABC.
prd.flags |= FLAG_TERMINATE;
}
// Inside the ray generation program:
RT_PROGRAM raygen()
{
// Setup primary ray and rtPayload for an iterative path tracer.
// ...
prd.state = STATE_START;
// ...
while (depth < MAX_PATH_LENGTH) // Limit this to the minimum number required to match the state expression.
{
// Potentially setup some more data per ray segment here.
// ...
rtTrace(root, ray, prd);
if (prd.state == STATE_MATCH_ABC || prd.state == STATE_MATCH_ABMiss)
{
// Found path which matches the order of A->B->C or A->B->Miss and can distinguish either.
break;
}
else if (prd.state == STATE_NO_MATCH)
{
// The path did not hit the objects in the desired order.
break;
}
// else continue the path.
++depth;
}
}
A brute force path tracer which could easily be changed to do that can be found in optixIntro_04 here:
https://github.com/nvpro-samples/optix_advanced_samples/tree/master/src/optixIntroduction
In an existing path tracer this wouldn’t need any additional ray type either. This can be implemented by tagging a ray as a special case inside some rtPayload field which controls the behaviour of the individual program domains, including early exits.
This could be handled with a special ray type and the minimal required program code if that is easier, but I normally try to reduce the amount of OptiX objects to the minimum necessary.