How does OptiX determine if the built in triangle intersection happens on the front face or the back face? How does the program determine which face is front or back?

Interesting, that seems to be missing in the OptiX documentation.

That’s defined by the triangle winding. The default front face is counter-clockwise winding in a right-handed coordinate system (OpenGL), resp. clockwise in a left-handed coordinate system (DirectX).

The optixGetHitKind() result can be flipped per instance with: OPTIX_INSTANCE_FLAG_FLIP_TRIANGLE_FACING

https://raytracing-docs.nvidia.com/optix7/api/html/group__optix__types.html#gadefef6077edef8bb8f4008d27d784745

There are three convenience functions over optixGetHitKind():

https://raytracing-docs.nvidia.com/optix7/api/html/group__optix__device__api.html#ga39492a97a6301dd6eb51c48e020a54cd

For custom primitives you have 7 bits you can set as hit kind in optixReportIntersection().

https://raytracing-docs.nvidia.com/optix7/api/html/group__optix__device__api.html#ga8ef53b8aca3f16a532142ee13f987c2a

Hmm, if our ray generation point is within the scene itself(opposed to a pixel on the screen) will the output of optixIsTriangleFrontFaceHit() change with respect to change in the raygen point?

Let’s say the triangle defines an infinite plane which is splitting space into two half-spaces.

Name the two spaces front and back consistently depending on the triangle vertex order, e.g. triangle (v0, v1, v2) has the unnormalized face normal cross(v1 - v0, v2 - v0) above the front face.

Define the ray origin to lie in exactly one of these two half-spaces.

Then the ray can only hit that infinite plane (and therefore the triangle lying in it) from that space’s side, and only if the ray direction is actually pointing to that plane.

Yes, but lets assume it is a recursive raytacing program. The ray hits the triangle on one side(side A) and when queried using optixTriangleFrontFaceHit() it gives us a value( let’s assume true). When a new ray hits from the opposite side(side B), this time, will the optixTriangleBackFaceHit() return True or will Front face return true?