Not sure what your intended use case is with that approach, but I would not architect a renderer that way.
1.) any_hit and closest_hit programs are per ray type already, so they behave differently if you have different program implementations per ray type.
Doing different things inside the intersection or attribute program “per ray type” is not going to work. That would require to identify the ray type dynamically and that would need to be done with data on the rtPayload which in turn is not accessible in the intersection or attribute program domains.
EDIT: I missed the rtCurrentRay field ray_type. See below.
2.) Depending on your material system, it could be possible to have only one Material per GeometryInstance while still supporting arbitrarily many BSDFs.
Take a look at my OptiX Introduction examples which demonstrate how to do that for a simple case.
Links to presentation video, slides and source code here:https://devtalk.nvidia.com/default/topic/998546/optix/optix-advanced-samples-on-github/
The block diagram at the end of the README page is really the whole renderers architecture.
(The OptiX Advanced Samples are not updated to support OptiX 6.0.0, yet. The CMake version detection needs changes due to the new library naming scheme using the fully qualified major.minor.micro version now.)
A single material index is selecting what the BSDF and parameters are, and while I store that material index at the GeometryInstance to reduce the number of unique Material nodes to two in the whole program (not counting the area light), that material index could be calculated any way you like as late as required inside the single any_hit or closest_hit programs per ray type, or if you’re adventurous even per ray.
This would completely remove the need for rtGeometryTrianglesSetMaterialIndices required to support multiple materials with GeometryTriangles. I’ve never used more than one Material per GeometryInstance.
From experience with my MDL-capable path tracer implementation using they same mechanisms, I can say that this is also applicable to much more complex material systems than shown in the OptiX Introduction samples.
Sidenote from your other thread:
Please do not use variables for the ray type IDs. Replace them with hardcoded defines since they should never change.
Unfortunately OptiX SDK examples were doing that in the past unnecessarily and esp. in OptiX 6.0.0 that should better not be done.