Please have a look into the OptiX documentation.
This chapter explains programs:
Have a deeper look into chapter 4.8.2 when anyhit and closest hit programs are invoked:
Then I would recommend you look through the OptiX Introduction examples (links below) and what you’re looking for is the rtIgnoreIntersection() function inside the anyhit functions for both the radiance and shadow ray types, used for texture based cutout opacity in example 07.
Your example is the anyhit program on a shadow ray type only, but for face culling inside the anyhit program you would need some more code to handle the rtIgnoreIntersection when hitting the culled face.
The faster method would be to change your intersection program to do something like this based on the code in
Then you do not even have a potential intersection which would reach the anyhit program which should be the faster face culling mechanism today.
// 1.) Example how to early-exit the intersection program for backface hits.
// Unoptimized version for clarity.
// Mind that the rtCurrentRay is in object space inside the intersection program domain.
// Calculate the triangle face normal in object space for counter clock winding triangle vertices.
const float3 nf = optix::normalize(optix::cross(v1 - v0, v2 - v0));
// If the dot product is positive the ray points into the same hemisphere as the face normal, means it's hitting a backface.
// Also remove edge-on cases with the equal comparison.
if (optix::dot(nf, theRay.direction) >= 0.0f)
return; // Do not calculate anything more for this triangle in the intersection program.
// 2.) Or simpler and slower by reusing the face normal calculated by the OptiX provided triangle intersection routine:
if (intersect_triangle(theRay, v0, v1, v2, n, t, beta, gamma))
// Front face hit and potential intersection with t inside the current [t_min, t_max] interval?
if (optix::dot(n, theRay.direction) < 0.0f && rtPotentialIntersection(t))
// ... calculate attributes and report intersection