Back face culling

I noticed in the documentation there’s an option to enable back face culling using enum RTUinitoptions.RTU_INITOPTION_CULL_BACKFACE
. My question is where (and how) do I use it? at thich point in the program do I activate it and in general what’s the preferred pipeline?

Thank in advance,

The rtuTraversal API functions are dead. Please ignore that optixu_traversal.h header.
That had been long superseded by the low-level OptiX Prime ray intersection API.

I would recommend to use the high-level OptiX ray casting API instead. Backface culling in current OptiX versions can be easily implemented inside the intersection or anyhit program domain. You would just need to check the sign of a dot product between the face normal and the ray direction to ignore a potential intersection.

I implemented it using the rtIgnoreIntersection program.

When I put it in the anyhit program, nothing happens, and when I put in the closest hit program I received the following error:

ERROR :Invalid context (Details: Function “_rtMaterialSetClosestHitProgram” caught exception: Validation error: rtIgnoreIntersection call is not allowed in _Z11closest_hitv_cp17 function with semantic type CLOSEST_HIT)

I see now in the documentation that that rtIgnoreIntersection is valid only in any-hit program…
So I think i’m missing something here…

What is the relationship between the anyhit and the closesthit programs? because in my closest hit program I paint the pixels while my anyhit consists only of

RT_PROGRAM void any_hit()
    prd_shadow.inShadow = true;

how can two functions be incharge of coloring pixels?

Please have a look into the OptiX documentation.
This chapter explains programs:
Have a deeper look into chapter 4.8.2 when anyhit and closest hit programs are invoked:

Then I would recommend you look through the OptiX Introduction examples (links below) and what you’re looking for is the rtIgnoreIntersection() function inside the anyhit functions for both the radiance and shadow ray types, used for texture based cutout opacity in example 07.
Your example is the anyhit program on a shadow ray type only, but for face culling inside the anyhit program you would need some more code to handle the rtIgnoreIntersection when hitting the culled face.

The faster method would be to change your intersection program to do something like this based on the code in
Then you do not even have a potential intersection which would reach the anyhit program which should be the faster face culling mechanism today.

// 1.) Example how to early-exit the intersection program for backface hits. 
// Unoptimized version for clarity.
// Mind that the rtCurrentRay is in object space inside the intersection program domain.
// Calculate the triangle face normal in object space for counter clock winding triangle vertices.
const float3 nf = optix::normalize(optix::cross(v1 - v0, v2 - v0));
// If the dot product is positive the ray points into the same hemisphere as the face normal, means it's hitting a backface.
// Also remove edge-on cases with the equal comparison.
if (optix::dot(nf, theRay.direction) >= 0.0f)
  return; // Do not calculate anything more for this triangle in the intersection program.

// 2.) Or simpler and slower by reusing the face normal calculated by the OptiX provided triangle intersection routine:
if (intersect_triangle(theRay, v0, v1, v2, n, t, beta, gamma))
  // Front face hit and potential intersection with t inside the current [t_min, t_max] interval?
  if (optix::dot(n, theRay.direction) < 0.0f && rtPotentialIntersection(t))
    // ... calculate attributes and report intersection