I cannot say if that has been solved in an earlier version between 4.1.0 and 5.1.0.
The problem had not been tested earlier and that is the last information I found on this.
It’s definitely worth to give OptiX 5.1.x a try because that contains various improvements over 5.0.1.
Depending on the scene complexity, the 1 vs 3000 GeometryInstance runtime performance impact is not too surprising on that small Maxwell board. There are already three newer GPU generations available. Acceleration structure traversal performance would be the culprit here, if those have not been under a single GeometryGroup, but it sounds you put them under one node and that’s most likley the root of the scene. Then I don’t know.
What acceleration structure builder did you use? When using Trbvh are these triangles and did you set the acceleration properties to trigger the specialized triangle builder?
I wasn’t implying to separate the GeometryInstance into 3000 individual ones when I said it might be possible to have fewer Material nodes.
I don’t know what your material system is, but if it’s something like various instances of different material parameters for a few different fundamental BSDF implementations, then that can be handled with a single material.
Instead of using rtReportIntersection(materialIndex) inside your intersection program to select one of your 3000 materials, make that materialIndex an attribute and use that inside a single closesthit program to select the material instance parameters from a context global buffer of structs, where each struct contains all necessary material parameters and BSDF indices to configure the material behaviour inside the closesthit program via a function table implemented via a buffers of bindless callable programs, which implement the BSDFs.
That is exactly how the OptiX Introduction example 07 is doing it, though per GeometryInstance.
Search these sources for “initMaterials” and “parMaterialIndex”:
Means depending on how the materialIndex inside your intersection program is calculated, it could have a completely different material per ray if you wanted while still always calling rtReportIntersection(0).
Normally you would store that materialIndex per triangle primitive as an additional attribute, for example by using uint4 triangle indices with the .w component being the material index.