Back face material assignments with RTgeometrytriangles

I’d like to use RTgeometrytriangles while still supporting separate front and back face material assignments (which trigger different closest-hit programs, not simply the same program with different parameters). If I’m reading the documentation correctly, this isn’t directly supported, but it seems like using multiple GeometryInstances and the RT_INSTANCE_FLAG_FLIP_TRIANGLE_FACING option could make it work. Is that the recommended method, or am I overlooking something better?
I’m currently on OptiX 6.5.

Having the same geometry defined twice to be able to have different materials on front and back faces would mean the BVH would have all AABBs for each primitive twice. So that will affect BVH traversal performance because in the worst case you have twice the number of intersection tests just to determine that half of the work was in vain. Similarly for instances of the same geometry at the same place.
In principle that would work together with face culling, but I wouldn’t do that.

With custom primitives you would be able to select the material shader to call inside the intersection program.
The rtReportIntersection function’s parameter selects which material to use.
That in turn requires multiple materials to be used on the GeometryInstance, which I would also not recommend.

For built-in GeometryTriangles that isn’t going to work this way, since you cannot change the intersection program.
There you can only provide one material index per triangle statically, explained here:
which means you cannot have different closest hit programs per face.

That in turn means you would need to have a different closest hit program architecture to be able to determine the final shader depending on the triangle face you’re hitting.
I’ve been architecting my renderers to be able to handle that case for a long time. (They are a stripped-down version of an own MDL-capable renderer which allows thin-walled geometry with different materials on front and back side.)

The OptiX 5.1.0 Advanced Samples (in OptiX Introduction) and my OptiX 7 Advanced Samples for which you find links in these sticky posts demonstrate one option to handle that:

My examples use a renderer architecture in which the BXDF closures are abstracted from the closest-/any-hit programs using direct callables.
Means there is only one closest hit program for all materials and what material type and parameters is applied can be determined per hit, means the granularity of the material assignment possible is per ray!

Here would be the code place where you could select different shaders per face side:
In that case that would simply require having two parMaterialIndex variables (resp. an int2) per GeometryInstance for front and back and selecting one of them
This would work with Geometry and GeometryTriangles primitives. Latter would require an additional attribute program to calculate the accessed vertex attributes. (The current example code will build with OptiX 6.5.0 already but doesn’t use GeometryTriangles. I do not use OptiX 5 or 6 versions anymore.)

Same for the OptiX 7 versions of the examples which are using built-in triangles.
While that code checks if the geometric normal and the negative ray direction point to the same hemisphere, there also exist the OptiX 7 device function optixGetHitKind() which can be used inside the hit programs to determine the primitive type and for built-in triangles which side was hit.

Of course care would need to be taken to only generate physically plausible materials. The renderer itself doesn’t care, for example, if the transparency on each face side is different.

In general I would highly recommend to port OptiX applications to OptiX 7 versions because that offers the more modern API, is much more flexible and generally faster.