Please have a look into the OptiX documentation.

This chapter explains programs:

http://raytracing-docs.nvidia.com/optix/guide/index.html#programs#programs

Have a deeper look into chapter 4.8.2 when anyhit and closest hit programs are invoked:

http://raytracing-docs.nvidia.com/optix/guide/index.html#programs#reporting-intersections

Then I would recommend you look through the OptiX Introduction examples (links below) and what you’re looking for is the rtIgnoreIntersection() function inside the anyhit functions for **both **the radiance and shadow ray types, used for texture based cutout opacity in example 07.

Your example is the anyhit program on a shadow ray type only, but for face culling inside the anyhit program you would need some more code to handle the rtIgnoreIntersection when hitting the culled face.

https://devtalk.nvidia.com/default/topic/998546/optix/optix-advanced-samples-on-github/

https://github.com/nvpro-samples/optix_advanced_samples/blob/master/src/optixIntroduction/optixIntro_07/shaders/anyhit.cu

The faster method would be to change your intersection program to do something like this based on the code in

https://github.com/nvpro-samples/optix_advanced_samples/blob/master/src/optixIntroduction/optixIntro_07/shaders/intersection_triangle_indexed.cu

Then you do not even have a potential intersection which would reach the anyhit program which should be the faster face culling mechanism today.

```
// 1.) Example how to early-exit the intersection program for backface hits.
// Unoptimized version for clarity.
// Mind that the rtCurrentRay is in object space inside the intersection program domain.
// Calculate the triangle face normal in object space for counter clock winding triangle vertices.
const float3 nf = optix::normalize(optix::cross(v1 - v0, v2 - v0));
// If the dot product is positive the ray points into the same hemisphere as the face normal, means it's hitting a backface.
// Also remove edge-on cases with the equal comparison.
if (optix::dot(nf, theRay.direction) >= 0.0f)
{
return; // Do not calculate anything more for this triangle in the intersection program.
}
// 2.) Or simpler and slower by reusing the face normal calculated by the OptiX provided triangle intersection routine:
if (intersect_triangle(theRay, v0, v1, v2, n, t, beta, gamma))
{
// Front face hit and potential intersection with t inside the current [t_min, t_max] interval?
if (optix::dot(n, theRay.direction) < 0.0f && rtPotentialIntersection(t))
{
// ... calculate attributes and report intersection
}
}
```